DOCUMENTED  BRIEFING 


RAND 


Strategic  Sourcing 

Measuring  and  Managing 
Performance 


Laura  H.  Baldwin,  Frank  Camm, 
Nancy  Y.  Moore 


Prepared  for  the 
United  States  Air  Force 


Project  AIR  FORCE 


Approved  for  public  release;  distribution  unlimited 


PREFACE 


This  documented  briefing  discusses  the  use  of  performance  metrics  in 
outsourcing  by  innovative  commercial  customers  and  providers  of  facility 
management  services.  It  is  part  of  a  series  of  documented  briefings  from  the 
Project  AIR  FORCE  study,  “Improving  Readiness  Through  Increased  Access 
to  Private  Sources  of  Support.”  The  project  was  sponsored  by  the  Deputy 
Chief  of  Staff  for  Plans  and  Programs. 

This  work  should  interest  those  concerned  with  innovative  ways  to 
outsource  support  activities — especially  those  charged  with  expanding 
outsourcing  in  the  Air  Force  and  the  Department  of  Defense. 

Work  on  this  topic  and  related  outsourcing  topics  continues  in  the 
Resource  Management  Program  of  Project  AIR  FORCE.  For  additional 
information,  please  contact  the  study  leader.  Dr.  Frank  Camm,  at  (202)  296- 
5000  x5261  or  camm@rand.org.  To  convey  comments  on  this  paper  to  the 
authors,  please  contact  Dr.  Camm  or  the  principal  author,  Dr.  Laura 
Baldwin,  at  (310)  393-0411  x6537  or  Laura_Baldwin@rand.org. 

Project  AIR  FORCE 

Project  AIR  FORCE,  a  division  of  RAND,  is  the  Air  Force  federally 
funded  research  and  development  center  (FFRDC)  for  studies  and  analysis.  It 
provides  the  Air  Force  with  independent  analyses  of  policy  alternatives 
affecting  the  development,  emplo3rment,  combat  readiness,  and  support  of 
current  and  future  aerospace  forces.  Research  is  performed  in  four  programs: 
Aerospace  Force  Development;  Manpower,  Personnel,  and  Training;  Resource 
Management;  and  Strategy  and  Doctrine. 
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SUMMARY 


This  documented  briefing  describes  the  ways  innovative  commercial 
customers  and  world-class  providers  of  facility  management  services  use 
performance  metrics  to  execute  their  sourcing  activities.  The  sourcing 
process  has  several  steps,  including  evaluating  alternative  sources  of 
provision,  choosing  a  provider,  and  managing  a  relationship  with  that 
provider.  A  strategic  sourcing  process  links  the  decisions  made  during  the 
sourcing  process  to  the  strategic  goals  of  the  customer  organization. 
Performance  metrics  play  many  different  roles  throughout  the  strategic 
sourcing  process.  They  help  a  customer  firm  decide  whether  to  outsource  an 
activity,  guide  the  source-selection  decision,  manage  the  customer/provider 
relationship,  and  enable  continuous  improvement  in  the  provision  of  services. 

Under  pressure  to  improve  performance  and  reduce  infrastructure 
expenditures,  the  Air  Force  is  reexamining  the  source  of  provision  of  many  of 
its  support  activities  such  as  base  operating  support  and  other  installation 
services.  These  activities  appear  to  represent  lower  risks  and  a  higher 
potential  for  performance  improvements  and  cost  savings  than  many  other 
candidates  for  outsourcing.  In  our  examination  of  commercial  uses  of 
performance  metrics  in  sourcing  activities,  we  focus  on  facility  management 
services  because  they  are  similar  to  these  Air  Force  support  activities. 

The  customer  and  provider  firms  that  we  interviewed  represent  the 
innovative  leading  edge  of  commercial  sourcing  practices  rather  than  typical 
commercial  practices.  Although  the  priorities  and  constraints  that  the  Air 
Force  faces  in  its  sourcing  activities  are  not  always  the  same  as  those  of 
innovative  commercial  firms,  the  practices  that  these  firms  use  to  choose  and 
implement  performance  metrics  in  their  sourcing  processes  should,  with 
appropriate  adaptations,  help  the  Air  Force  reduce  its  oversight  burden  and 
improve  the  results  of  its  sourcing  activities. 

Roles  of  Performance  Metrics  in  the  Sourcing  Process 

Customers  and  providers  choose  different  performance  metrics  for 
different  purposes  throughout  the  sourcing  process.  A  customer  first  uses 
metrics  to  help  decide  whether  to  outsource  an  activity.  If  it  decides  to 


outsource  the  activity,  the  customer  uses  metrics  to  guide  the  selection  of  an 
external  provider.  Once  the  customer  has  chosen  a  provider,  the  customer 
and  provider  each  use  metrics  to  help  manage  their  relationship.  Finally, 
parallel  to  customers’  sourcing  activities,  providers  use  metrics  to  promote 
and  enable  continuous  improvement  in  the  provision  of  services.  Our 
primary  focus  is  on  how  firms  use  metrics  to  manage  the  customer/provider 
relationship. 

Choosing  the  Appropriate  Set  of  Metrics 

In  managing  their  relationship,  a  customer  and  a  provider  jointly  choose 
metrics  that  they  believe  will  support  the  corporate  goals  of  the  customer 
organization.  Such  metrics  align  the  provider’s  priorities  with  those  of  its 
customer.  Customers  tend  to  track  output-oriented  metrics,  whereas 
providers  may  track  additional  process-oriented  metrics.  Both  qualitative 
and  quantitative  metrics  are  common. 

We  discerned  several  patterns  in  the  choice  of  metrics  by  customers  and 
providers.  Customers  tend  to  focus  on  a  few  key  metrics  to  easily  convey  to 
providers  those  dimensions  of  performance  that  are  most  important. 
Customers  and  providers  refine  the  set  of  metrics  used  to  manage  their 
relationships  throughout  the  duration  of  those  relationships.  Metrics  evolve 
as  focus  and  needs  change  so  that  they  continue  to  add  value.  To  preserve 
the  flexibility  to  adjust  metrics  over  time,  many  customers  separate  the 
specific  metrics  they  will  use  to  manage  their  relationships  from  their 
contracts  with  providers. 

Although  we  observed  diversity  in  the  specific  choices  of  metrics  used  to 
manage  customer/provider  relationships,  several  topic  areas  were  common  to 
many  of  the  firms  we  interviewed.  Financial  and  customer  satisfaction 
metrics  and  quantitative  metrics  targeting  the  performance  of  specific 
services  (including  safety)  were  observed  most  often.  Many  firms  also  chose 
metrics  that  reflect  their  concern  for  human  resources  issues  and  special 
interests  such  as  socioeconomic  concerns. 

Collecting  Data  and  Generating  Metrics 

Customers  and  providers  use  several  methods  to  collect  data  to  generate 
the  metrics  that  they  track  to  manage  their  relationships.  Customer  surveys 


-vui- 


are  used  to  collect  customer  satisfaction  data.  Management  information 
systems,  such  as  those  associated  with  centralized  call  centers  and  bar-code 
scanners,  are  used  to  collect  detailed  data  about  operations.  Customers  and 
providers  also  perform  financial  and  operational  audits,  or  bring  in  third- 
party  auditors,  to  verify  performance  and  costs. 

Many  firms  expressed  frustration  with  data  accuracy.  Customer  firms 
can  take  a  variety  of  steps  to  verify  the  accuracy  of  metrics  generated  by  their 
providers — spot-checking  work,  collecting  samples  of  data,  and  auditing 
providers’  data  collection  and  metrics  generation  processes. 

Managing  Performance  and  Cost  Against  Goals 

Customers  and  providers  set  mutually  agreeable  goals  for  metrics  that 
reflect  both  short-run  and  long-run  desired  levels  of  performance  and  cost  for 
the  customer  organization.  Benchmarking  studies  often  are  used  to  set 
feasible  goals.  However,  because  of  the  difficulty  of  finding  “apples-to-apples” 
comparisons,  customers  and  providers  stressed  the  need  to  understand 
sources  of  variances  in  performance  and  costs  across  sites  and  companies  to 
ensure  that  goals  are  based  on  process  or  technological  innovations,  rather 
than  differences  in  facilities’  characteristics  or  uses  or  in  levels  of  services. 

Once  goals  are  in  place,  customers  and  providers  use  both  informal  and 
formal  reviews  to  facilitate  communication  about  performance.  Project 
managers  for  both  organizations  tend  to  meet  informally  daily  or  weekly  to 
discuss  ongoing  work.  Formal  reviews  occur  less  often,  involve  additional 
corporate  leaders,  and  are  used  to  monitor  performance  along  specified 
dimensions. 

Both  customer  and  provider  firms  noted  that  it  is  costly  to  generate 
metrics  and  track  performance  and  cost.  Ultimately,  these  monitoring  costs 
are  paid  by  customers;  therefore,  providers  advise  their  customers  to  request 
and  track  only  those  metrics  that  add  net  value  to  their  organizations. 

Implications  for  the  Air  Force 

The  sourcing  process  is  not  new  to  the  Air  Force;  however,  many  of  the 
practices  discussed  here  differ  dramatically  from  the  Air  Force’s  traditional 
approach  to  sourcing.  Nonetheless,  the  practices  of  innovative  commercial 
firms  are  broadly  compatible  with  current  Air  Force  efforts  to  extend  the 
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benefits  of  acquisition  reform  to  services  acquisition.  Effective  metrics  lie  at 
the  heart  of  this  effort.  The  Air  Force  can  adapt  the  practices  discussed  here 
to  improve  the  effectiveness  of  services  acquisition  and,  in  the  long  run,  move 
beyond  its  current  plans  toward  even  more  innovative  commercial  acquisition 
practices  that  allow  for  greater  performance  and  cost  improvements  and  that 
encourage  more  of  the  best  commercial  providers  to  want  to  do  business  with 
the  Air  Force.  However,  the  Air  Force  cannot  accomplish  this  alone.  Broader 
public  policy  will  have  to  advance  to  accommodate  all  the  innovative  practices 
discussed  here. 

First,  the  Air  Force  can  use  metrics  to  focus  its  limited  sourcing 
resources  on  those  activities  that  appear  to  offer  the  highest  potential  returns 
in  performance  and  cost  improvements.  Audits  that  baseline  internal 
performance  and  cost  and  benchmarking  studies  that  identify  external 
capabilities  can  facilitate  this  process. 

Second,  if  the  Air  Force  chooses  to  conduct  a  competition  for  provision  of 
services,  it  can  use  metrics  to  focus  the  source  selection  decision  on  those 
aspects  of  performance  and  cost  that  best  support  the  goals  of  the  Air  Force 
customer  organization. 

Third,  the  Air  Force  can  use  metrics  to  better  manage  its  relationships 
with  external  providers.  For  example,  output-oriented,  rather  than  process- 
oriented,  metrics  give  providers  the  freedom  to  innovate.  New  Air  Force 
policy  provides  installations  the  flexibility  to  update  these  metrics  outside  of 
formal  contract  modifications  to  reflect  the  changing  needs  of  rotating  wing 
commanders  so  that  providers  stay  focused  on  current  priorities.  Metrics  also 
can  facilitate  communication  between  customer  and  provider  organizations, 
which  can  lead  to  more  responsive  service.  Good  metrics  are  powerful  tools 
that  allow  a  small  group  of  project  managers  to  track  the  important 
dimensions  of  provider  performance.  Thus,  if  the  Air  Force  uses  metrics  that 
support  its  strategic  goals,  sets  aggressive  goals  for  the  metrics,  and  manages 
performance  through  open,  frequent  communication  and  problem  solving 
supplemented  by  longer-term  efforts  to  improve  performance,  the  experiences 
of  innovative  commercial  firms  suggest  that  the  Air  Force  can  build  mutual 
trust  with  its  providers  and  thereby  dramatically  reduce  the  manpower  and 
other  costs  associated  with  its  quality  assurance  program. 
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Finally,  independent  of  outsourcing,  the  Air  Force  can  use  performance 
metrics  to  improve  the  performance  of  its  organic  provider  organizations.  In 
particular,  cross-site,  as  well  as  Air  Force/extemal,  benchmarking  studies  can 
be  used  to  promote  sharing  of  best  practices  and  to  set  performance  goals. 
However,  good  metrics  and  aggressive  goals  will  not  be  effective  unless 
performance  is  tracked  and  personnel  are  held  accountable  for  meeting  goals. 
The  Air  Force  may  benefit  from  investing  in  a  commercial  management 
information  system  or  outsourcing  data  collection  to  generate  desired 
information  about  service  outputs  and  processes. 
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1.  INTRODUCTION 


Strategic  Sourcing:  Measuring  and 
Managing  Performance 


Laura  H.  Baldwin,  Frank  Camm,  Nancy  Y.  Moore 


RAND  Project  AIR  FORCE 


This  documented  briefing  describes  Project  AIR  FORCE  research  on  the 
ways  innovative  customers  and  providers  of  facility  management  services  use 
performance  metrics  to  help  execute  their  sourcing  activities.  Performance 
metrics  play  many  different  roles  throughout  the  sourcing  process,  in  which  a 
customer  decides  whether  to  outsource  or  keep  an  activity  or  service  in-house, 
chooses  a  source,  and  manages  its  relationship  with  the  chosen  provider, 
including  the  pursuit  of  continuous  improvement  in  the  provision  of  the 
activity  or  service.  Under  performance-based  services  acquisition,  each  of 
these  issues  arises  in  the  Air  Force  sourcing  process.  The  priorities  and 
constraints  that  the  Air  Force  faces  in  its  sourcing  process  are  not  always  the 
same  as  those  faced  by  innovative  commercial  customers,  but  the  practices 
that  innovative  commercial  firms  use  to  choose  and  implement  performance 
metrics  in  their  sourcing  processes  should,  with  appropriate  adaptations, 
help  the  Air  Force  reduce  its  oversight  burden  and  improve  the  results  of  its 
sourcing  activities. 
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This  briefing  is  the  third  in  a  series  of  documents  based  on  our 
structured  interviews  with  “world-class”  commercial  providers  of  facility 
management  services  and  with  commercial  firms  that  buy  such  services  from 
these  providers.  The  first  draft  document  provides  an  overview  of  our 
findings,  touching  on  a  broad  range  of  topics  including  (1)  why  firms  choose  to 
outsource,  (2)  the  outsourcing  process,  (3)  how  firms  bundle  activities  for 
outsourcing,  (4)  source  selection,  (5)  contract  design,  (6)  the  transition 
process,  and  (7)  managing  the  customer/provider  relationship.  A  second  draft 
document  details  how  innovative  firms  bundle  activities  to  be  outsourced  and 
presents  a  process  for  doing  so  that  other  organizations  should  find  useful. ^ 


^  Copies  of  these  two  draft  documents  can  be  obtained  from  Dr.  Nancy  Moore,  subject 
to  approval  from  the  research  sponsor.  She  can  be  reached  via  email  at 
N  ancy_Moore@rand.org. 
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1  strategic  Sourcing  Can  Help  the  Air  Force  1 
1  Achieve  Better  Sourcing  Results  I 
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Metrics  play  important  roles  throughout 
execution  of  strategic  sourcing 
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Facing  strong  pressure  to  improve  performance  and  reduce 
expenditures  on  infrastructure,  the  Air  Force  has  embarked  upon  a  multiyear 
plan  to  re-examine  the  source  of  provision  of  many  of  its  support  activities. 
RAND’s  recent  research  on  outsourcing,  privatization,  and  recontracting  has 
been  oriented  toward  defining  a  strategic  sourcing  process  that  the  Air  Force 
can  use  to  achieve  the  best  possible  results  from  its  sourcing  activities.  Such 
a  strategic  sourcing  process  links  decisions  made  during  the  sourcing  process 
to  the  strategic  goals  of  the  customer  organization.  As  we  will  see, 
performance  metrics  play  key  roles  throughout  the  execution  of  strategic 
sourcing. 
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During  Fiscal  Years  (FY)  1996-1997,  RAND  examined  innovative 
sourcing  practices  of  commercial  customers  and  providers  of  both  third-party 
logistics  services  and  facility  management  services. 

During  FY  1996,  we  examined  how  innovative  commercial  firms  decide 
between  in-house  and  external  provision  of  services.  This  work  is  based,  in 
part,  on  visits  with  selected  customers  and  providers  of  logistics  services. 
During  FY  1997,  we  focused  on  detailed  aspects  of  the  sourcing  process  in 
visits  with  selected  customers  and  providers  of  facility  management  services. 

We  emphasize  that  the  sourcing  activities  of  the  firms  we  interviewed 
do  not  represent  standard  commercial  practices.  Rather,  these  firms  were 
chosen  because  of  their  innovative  approaches  that  have  led  to  their 
recognition  for  successful  sourcing  practices  (e.g.,  improved  performance, 
more  control  and  flexibility,  lower  costs  of  services).  In  other  words,  these 
customers  and  suppliers  are  defining  the  leading  edge  of  commercial  sourcing 
practices. 

To  find  the  most  innovative  sourcing  practices,  we  searched  for 
exemplar  firms  that  were  identified  by  their  peers  as  being  the  most 
progressive  customers  or  best  providers  in  the  markets  for  third-party 
logistics  and  facility  management  services. 
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We  began  by  identif5ring  an  initial  group  of  customer  and  provider  firms 
through  a  review  of  the  trade  literature,  contacts  at  trade  organizations,  and 
attendance  at  conferences.  To  progress  from  this  initial  set  of  firms  to  the 
final  group  of  firms  that  we  visited,  we  asked  these  firms  a  series  of 
questions.  We  asked  innovative  customers  who  their  providers  were  and 
which  other  leading  firms  they  had  considered  choosing;  we  asked  leading 
providers  who  their  competitors  were;  and  we  asked  leading  providers  which 
of  their  customers  were  engaged  in  the  most  innovative  outsourcing  practices. 
We  selected  30  companies  that  are  customers  or  providers  of  third-party 
logistics  or  facility  management  services,  and  we  made  41  site  visits, 
including  multiple  divisions  of  nine  of  the  companies.^  We  also  visited  four 
Air  Force  installations  that  have  outsourced  large  bundles  of  support 
services. 

During  site  visits  to  both  customer  and  provider  locations,  we 
interviewed  key  managers  about  their  outsourcing  experiences,  using  a 
structured  questionnaire  that  we  provided  in  advance.  Of  primary  interest 
were  the  practices  and  experiences  of  these  firms  concerning  how  they  (1) 
bundled  services  to  be  outsourced,  (2)  selected  one  or  more  providers,  (3)  set 
up  the  initial  contract(s),  (4)  transitioned  from  an  in-house  workforce  to  an 
external  one,  (5)  managed  the  customer/provider  relationship,  and  (6)  used 
performance  metrics  throughout  this  process. ^ 

This  document  focuses  on  the  use  of  performance  metrics  in  the 
strategic  sourcing  process,  and  it  draws  upon  our  site  interviews  with 
customers  and  providers  of  facility  management  services,  presentations  at 
trade  association  meetings,  and  the  business  literature.  The  use  of 
performance  metrics  in  the  strategic  sourcing  process  was  not  a  primary 
focus  area  during  our  interviews  with  customers  and  providers  of  third-party 
logistics  services;  however,  the  principal  ideas  and  practices  are  quite  similar 
across  logistics  and  facility  management  services. 

2  Twenty-two  of  these  organizations  purchase  and  nineteen  provide  facility 
management  or  logistics  services.  We  visited  one  or  more  customers  for  each  of  the 
providers. 

3  The  questions  addressing  the  use  of  performance  metrics  are  contained  in  Appendix 
A  of  this  document. 


-5- 


We  focus  on  facility  management  services  because  of  their  similarity 
with  base  operating  support  (BOS)  and  other  services  that  are  provided  at 
Air  Force  installations.  There  is  high-level  Air  Force  interest  in  reexamining 
the  source  of  provision  of  these  kinds  of  activities,  partly  because  these 
activities  represent  lower  risks  than  many  other  prospective  candidates  for 
outsourcing  and  have  a  high  potential  for  improved  performance  and  cost 
savings. 

The  lower  risk  associated  with  outsourcing  these  activities  stems  from 
several  factors.  First,  installation  services  and  base  maintenance  activities 
(e.g.,  grounds,  custodial,  and  building  maintenance)  are  often  only  indirectly 
related  to  the  primary  missions  of  the  Air  Force.  Second,  the  robust 
commercial  market  for  many  of  these  services  and  activities  suggests  that  the 
Air  Force  would  have  many  choices  for  providers,  so  that  no  one  provider 
could  exert  monopoly  pressure.  Third,  many  leading  commercial  firms  are 
expanding  their  outsourcing  activities  in  the  facilities  support  area, 
suggesting  that  they  have  identified  ways  to  mitigate  most  of  the  risks 
associated  with  outsourcing  these  activities.  Finally,  most  BOS  and  related 
support  services  have  already  been  successfully  outsourced  at  one  or  more  Air 
Force  installations. 

The  potential  for  performance  improvement  in  BOS  and  related  support 
services  stems  from  the  likely  gaps  between  the  capabilities  of  leading 
commercial  providers  and  those  of  in-house  Air  Force  providers.  Such  gaps 
stem  from  differing  levels  of  investments  in  capital  (e.g.,  management 
information  systems)  and  training  of  both  managers  and  technicians. 
Commercial  facility  management  providers  justify  making  much  larger 
investments  because  these  activities  are  their  core  competencies.^ 

■^he  concept  of  “core  competencies”  was  developed  by  Prahalad  and  Hamel  (1990)  and 
is  now  peiwasive  in  the  business  management  literature.  A  firm  s  core  competencies  are 
limited  to  a  few  activities  that  are  most  critical  to  the  future  success  of  the  organization. 
Generally,  these  activities  should  never  be  outsourced  (although  some  firms  have  outsoimced 
a  core  activity  and  then  later  brought  it  back  in-house).  See  Pint  and  Baldwin  (1997)  for  an 
overview  of  the  business  literatiire  on  this  topic. 

The  Department  of  Defense’s  definition  of  “core”  activities  should  not  be  confused  with 
“core  competencies”  as  defined  in  the  business  literature.  These  two  concepts  evolved 
independently  of  one  another  and,  in  principle,  could  differ.  However,  both  refer  to  activities 
that  are  deemed  so  important  to  the  organization  that  they  should  not  be  outsoxirced. 
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Although  commercial  estimates  of  cost  savings  associated  with 
outsourcing  facility  management  services  vary  widely,  every  customer  that 
we  spoke  with  was  confident  that  savings  do  exist Because  of  the  large 
number  of  commercial  activities  billets  currently  associated  with  BOS  and 
related  support  areas,®  the  potential  exists  for  the  Air  Force  to  realize 
enormous  savings  through  sourcing  studies^  Further,  existing  contracts 
could  be  rebundled  into  larger,  integrated  service  contracts  that  are  likely  to 
yield  additional  savings  from  economies  of  scale  and  scope  and  reduced 
transactions  costs. 


®The  Morrison  Knudsen  Corporation,  a  leading  provider  of  facility  management 
services,  estimates  initial  savings  between  5  and  24  percent  and  ongoing  annual  savings 
from  continuous  improvement  between  3  and  22  percent.  (Presentation  at  the  1997 
Tradeline  Conference,  San  Diego,  CA.) 

®  The  Air  Force’s  Commercial  Activities  Inventory  Reporting  System  (CAIRS) 
database  (9/30/95)  indicates  that  there  are  17,000  civilian  billets,  49,000  miUtary  billets,  and 
27,000  Contractor  Manpower  Equivalents  (CMEs)  in  installation  services  and  base 
maintenance. 

^The  Air  Force  used  20  percent  in  the  past,  and  25  percent  more  recently,  to  estimate 
expected  savings  from  sourcing  competitions.  Although  we  have  not  seen  a  document 
describing  the  methodology  used  to  derive  these  estimates,  we  believe  that  they  are  based  in 
part  on  estimates  of  initial  savings  from  prior  sourcing  studies.  In  a  study  of  Navy 
competitions  for  provision  of  commercial  activities,  Marcus  (1993)  found  an  average  savings 
of  30  percent.  However,  these  estimates  of  savings  are  thought  to  be  exaggerated  as  a  result 
of  a  lack  of  information  about  (a)  the  mihtary/civilian  personnel  mix  prior  to  sourcing  studies, 
(b)  the  civilian  pay  grades  associated  with  an  activity  before  and  after  a  study,  and  (c)  the 
scope,  quality,  and  type  of  work  performed  before  and  after  the  study.  See  Baldwin,  Camm, 
Keating,  and  Pint  (1998)  and  Robbert,  Gates,  and  Elhott  (1997)  for  a  more  detailed 
discussion  of  historical  savings  associated  with  military  sourcing  studies. 
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We  visited  world-class  providers  of  a  wide  range  of  facility  management 
services,  including  food  services,  engineering  and  construction,  controls,  and 
property  management.  Their  most  innovative  customers  include  electronics 
firms,  manufacturers  of  high-tech  equipment,  fast-food  corporations,  banks, 
and  defense  contractors. 

Many  of  these  providers  do  not  currently  do  business  with  the 
Department  of  Defense  (DoD).  These  innovative  sourcing  practices  have 
evolved  almost  entirely  within  the  commercial  sector. 
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Key  Points  to  Take  Away  with  You 


•  Performance  metrics  help  ensure  that  sourcing 
activities  promote  strategic  corporate  goals 

•  A  relationship  based  on  mutual  trust  results  in  more 
flexible  and  responsive  service 


•  The  choice  of  metrics  depends  on  the  purpose  and 
the  user 

•  There  is  diversity  across  firms  in  the  choices  of 
specific  performance  metrics,  even  within  one 
functional  area 


RAND  Project  AIR  FORCE 


Our  research  contains  four  overarching  themes.  The  first  is  that 
innovative  customers  of  facility  management  services  use  performance 
metrics  to  help  them  ensure  that  the  results  of  their  sourcing  activities 
support  their  strategic  corporate  goals.  This  begins  with  the  decision  to  stay 
with  the  in-house  provider  organization  or  seek  an  external  provider  of 
services.  A  customer  firm  uses  performance  metrics  to  compare  the  cost  and 
performance  of  its  in-house  organization  with  that  of  external  providers.  If 
external  provision  proves  to  be  more  effective,  the  customer  then  uses  metrics 
to  help  identify  the  provider(s)  best  able  to  meet  the  customer’s  needs. 

Finally,  performance  metrics  help  customers  manage  their  relationships  with 
providers  by  conununicating  expectations  and  providing  a  vehicle  to  track 
results. 

Metrics  often  reflect  a  balance  between  the  customer  firm’s  short-run 
and  long-run  strategic  priorities.  Consequently,  metrics  emphasize  near- 
term  performance/cost  objectives  as  well  as  longer-term  objectives  for  process 
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improvements  that  will  help  meet  the  customer’s  (potentially  unknown) 
future  needs.s 

Second,  performance  metrics  play  an  important  role  in  developing  a 
relationship  between  the  customer  and  provider  that  is  based  on  mutual 
trust.  Such  a  relationship  results  in  more  flexible,  responsive  service  from 
the  provider.  Keys  to  building  such  a  relationship  are  (a)  to  align  metrics 
with  those  aspects  of  performance  and  cost  that  are  most  important  to  the 
customer,  (b)  to  promote  open  communication  between  the  customer  and 
provider  organizations,  (c)  to  adjust  metrics  and  goals  as  customer  priorities 
change,  and  (d)  to  include  higher  levels  of  management  from  both  the 
customer  and  provider  organizations  in  periodic,  formal  performance  reviews. 
By  treating  the  provider  as  a  member  of  the  customer’s  management  team 
and  communicating  to  the  provider  changes  in  workscope,  budgets,  or 
expectations  in  a  timely  way,  a  customer  can  gain  the  trust  of  its  provider. 
Providers  gain  the  trust  of  their  customers  by  delivering  the  level  of  services 
promised. 

Third,  there  is  no  magic  set  of  metrics  that  fits  every  sourcing  need. 

The  choice  of  metrics  for  both  the  customer  and  provider  varies  throughout 
the  sourcing  process  because  information  needs  change  over  time.  For 
example,  when  deciding  whether  to  outsource,  customers  may  use  fairly 
broad  metrics  to  determine  the  relative  performance,  cost,  and  risks 
associated  with  in-house  versus  external  provider  organizations  (e.g.,  cost  of 
facility  management  services  per  square  foot).  If  a  decision  is  made  to 
outsource,  customers  may  seek  more-detailed  metrics  to  evaluate  the 
performance  and  cost  of  different  providers  along  several  key  dimensions  of 
services  (e.g.,  cost  of  specific  services  per  square  foot,  safety  history,  and 
employee  turnover).  Then,  during  a  relationship,  customers  and  providers 
may  track  a  variety  of  metrics  covering  many  dimensions  of  performance  and 
cost  at  varying  levels  of  detail  (e.g.,  number  of  unscheduled  maintenance  calls 
by  service,  average  cost  associated  with  unscheduled  maintenance,  or 
percentage  of  preventive  maintenance  performed  on  schedule). 

®See  Kaplan  and  Norton  (1996)  for  further  discussion  of  choosing  metrics  that  reflect 
the  many  dimensions  of  corporate  strategy. 
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The  choice  of  metrics  will  also  vary  between  the  parties  in  the  sourcing 
process — each  seeking  information  relevant  to  its  own  goals.  For  example,  in 
managing  their  relationship,  customers  tend  to  track  those  dimensions  of 
performance  that  indicate  whether  they  are  receiving  the  service  that  they 
need,  whereas  providers  track  indicators  of  whether  their  processes  are 
generating  appropriate  results.  Similarly,  managers  at  different  levels 
within  the  customer  firm  are  often  interested  in  metrics  that  depict 
information  that  has  been  aggregated  to  var5dng  degrees.  Whereas  the 
project  manager  may  be  interested  in  tracking  the  cost  associated  with  a 
specific  service  such  as  employee  moves,  corporate  managers  may  wish  to 
track  only  total  facilities  costs. 

Fourth,  there  is  diversity  across  firms  in  the  choice  of  performance 
metrics  to  provide  information  within  a  given  part  of  the  sourcing  process. 
For  example,  each  customer  firm  we  interviewed  chose  a  different  set  of 
metrics  to  select  its  provider.  In  large  part,  these  differences  reflect  the  fact 
that  the  facility  management  goals  of  the  customer  and  provider  in  each 
relationship  are  related  to  the  unique  strategic  goals  of  the  customer. 

We  believe  that  the  diversity  in  the  choices  and  uses  of  metrics  across 
customer  and  provider  firms  also  can  be  attributed  to  the  continuing 
evolution  of  these  innovative  practices.  In  fact,  many  customers  appear  to  be 
conducting  informal  sourcing  experiments,  trying  bold  new  practices  to  see 
whether  they  can  be  comfortably  applied  more  broadly.  Customers  told  us 
that  part  of  their  motivation  for  trying  new  sourcing  practices  is  to  encourage 
expansion  in  the  provider  market.  Similarly,  providers  are  trying  new 
practices  to  encourage  customers  to  feel  comfortable  outsourcing  more  of 
their  facility  management  services. 

This  diversity  raises  the  question  of  how  to  interpret  the  practices 
discussed  in  this  document.  Many  practices  represent  trends  that  are  either 
widely  accepted  ways  of  doing  business  within  this  community  of  innovative 
firms  or  activities  that  many  firms  said  they  plan  to  pursue  in  the  future.  In 
those  cases  where  firms  spanned  the  spectrum  of  approaches  to  a  given 
sourcing  activity,  we  chose  to  present  the  most  aggressive  practices  that 
represent  the  cutting  edge. 
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•  The  many  roles  of  performance  metrics  in  the 
sourcing  process 

•  Choosing  the  appropriate  set  of  metrics 

•  Collecting  data  and  generating  metrics 

•  Managing  performance  and  cost  against  goals 

•  Implications  for  the  Air  Force 


II 

I 


■  RAND  Project  AIR  FORCE 
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The  remainder  of  this  briefing  is  divided  into  five  chapters.  In  Chapter 
2,  we  examine  the  various  ways  that  customers  and  providers  use 
performance  metrics  in  the  sourcing  process.  Chapter  3  discusses  how 
customers  and  providers  choose  metrics  that  are  tailored  to  their  specific 
sourcing  situations.  In  Chapter  4,  we  look  at  how  commercial  firms  collect 
data  and  generate  the  metrics  that  they  use.  Chapter  5  examines  how  firms 
choose  goals  for  their  metrics  and  then  manage  the  provider’s  performance 
and  cost  against  those  goals.  We  conclude  in  Chapter  6  by  discussing  how  the 
Air  Force  might  use  performance  metrics  more  effectively  to  achieve  better 
results  from  its  sourcing  activities. 

Appendix  A  contains  the  portions  of  our  customer  and  provider 
questionnaires  that  address  the  use  of  performance  metrics.  Appendix  B 
contains  examples  of  performance  metrics  used  by  customers  and  providers  of 
facility  management  services. 
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2.  ROLES  OF  PERFORMANCE  METRICS 


Performance  Metrics  Play  Different  Roles 
Throughout  the  Sourcing  Process 


Customers  and  providers  use  performance  metrics  for  many  different 
things  throughout  the  sourcing  process.  The  choice  of  metrics  varies 
depending  on  where  the  parties  are  in  the  sourcing  process  (e.g.,  the 
make/buy  decision  vs.  the  source  selection  decision)  and  the  user  of  the 
information  (e.g.,  customer  vs.  provider). 

A  customer  first  uses  performance  metrics  to  help  decide  whether  to 
outsource.  If  it  does  decide  to  outsource,  the  customer  then  uses  metrics  to 
guide  the  source  selection  decision.  Once  a  customer  has  chosen  a  provider, 
the  customer  and  provider  each  use  metrics  to  help  manage  their 
partnership.  Finally,  parallel  to  customers’  sourcing  processes,  providers  use 
metrics  to  promote  and  enable  continuous  improvement  in  the  provision  of 
services. 

The  next  charts  discuss  each  of  these  roles. 
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Performance  Metrics  Help  Firms  Determine 

Whether  to  Outsource  a  Noncore  Activity 


•  Metrics  are  used  to 

-  benchmark  in-house  performance  and  costs 
against  potentiai  providers 

-  convey  the  objectivity  of  the  decision  process 

•  A  firm  mav  outsource  when  an  external  provider  can 
deliver 

-  equivalent  performance  at  lower  cost 

-  better  performance  for  no  more  than  in-house 
cost 


RANDProiect  AIR  FORCE! 


When  faced  with  the  decision  of  whether  to  outsource  an  activity  that  is 
not  a  core  competency,  customer  firms  can  use  metrics  to  compare  the 
performance  and  cost  of  their  internal  provider  organization  with  those  of 
external  providers.  When  armed  with  this  information,  customers  can  make 
the  decision  that  best  supports  their  strategic  corporate  goals.  Such 
information  about  the  internal  organization  also  provides  a  baseline  against 
which  future  performance  and  cost  can  be  compared. 

In  addition,  several  customer  project  managers  said  that  the  use  of 
objective  measures  in  the  decision  process  helped  convince  their  senior 
leadership  that  their  recommendation  for  internal  or  external  provision  of 
facilities  services  was  in  the  best  interest  of  the  firm — ^i.e.,  that  it  supported 
corporate  goals  and  provided  the  best  value  of  services  for  the  money.  As  we 
discuss  below,  the  support  of  high-level  management  is  an  integral  part  of  a 
successful  customer/provider  relationship,  whether  the  provider  is  an 
internal  or  external  organization. ^ 

^  When  outsourcing  was  warranted,  these  objective  measures  also  helped  make  the 
case  to  affected  employees  and  those  in  other  parts  of  the  customer  firm  that  the  decision 
process  was  fair. 
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Grenerally,  the  customer  organizations  we  visited  viewed  the  ability  of 
an  external  provider  to  provide  either  equivalent  performance  at  a  lower  cost 
or  better  performance  for  no  more  than  the  cost  of  the  in-house  organization 
as  a  necessary  condition  for  outsourcing.  However,  meeting  one  or  both  of 
these  criteria  did  not  guarantee  a  decision  to  outsource. 

Customers  also  use  other,  less  easily  quantifiable  criteria,  such  as  the 
risk  of  catastrophic  failure  to  perform  or  the  risks  associated  with  loss  of 
control,  to  help  decide  whether  to  outsource  a  given  activity. 

Most  customers  we  interviewed  did  not  adequately  measure  the 
performance  of  their  in-house  provider  organization  prior  to  being  faced  with 
the  outsourcing  decision.  Therefore,  they  had  to  implement  an  internal 
measurement  process  prior  to  making  the  comparison  with  external 
sources.io  Ideally,  in-house  provider  organizations  use  performance  metrics 
to  continually  improve  internal  processes  independent  of  deciding  whether  to 
outsource.  Without  such  measurements,  in-house  providers  may  not  know 
what  or  how  to  improve.  However,  customer  time  and  resource  constraints 
do  not  always  allow  for  this. 


Some  firms  used  internal  resources  to  implement  this  meastirement  process;  others 
used  consultants. 
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1  Performance  Metrics  Help  Customers  i 

1  Choose  Among  Potential  Providers  | 

Evaluation  categories  include 

: 

•  Total  cost 

b 

•  Quality  of  service 

.  Safety 

- 

•  HR  policies 

•  Use  of  technology 

•  Training 

•  Financial  stability 

•  Special  interests 

1 

m 

w 

1 

9 

. . . . . . . .  ,  ,  . . H 

Once  the  customer  has  decided  to  outsource  provision  of  a  service  or  a 
set  of  services,  the  customer  can  use  performance  metrics  and  associated 
weights  to  pick  the  provider  or  group  of  providers  that  best  meets  its  strategic 
objectives.  Customers  use  metrics  to  evaluate  providers  along  many 
dimensions  of  performance.  Although  no  two  customers  chose  the  exact  same 
criteria  to  evaluate  providers,  many  customers  were  interested  in  similar 
dimensions  of  performance.  The  above  chart  lists  the  categories  of 
performance  that  the  customers  we  interviewed  cited  most  often. 

Total  cost  includes  the  price,  customer  management  costs,  and  any 
implications  for  facility  life-cycle  costs  associated  with  a  given  level  of 
services.  The  quality  of  service  can  encompass  many  dimensions  of 
performance  such  as  reliability,  responsiveness  of  the  provider  to  changing 
customer  needs,  and  reputation  based  on  past  performance  for  other 
customers.  Safety  records  and  human  resources  policies  reflect  how  the 
provider  organization  treats  its  employees.ii  These  metrics  are  especially 

llThese  metrics  also  provide  information  that  is  useful  in  the  “due  diligence”  process. 
Customer  firms  use  safety  records  and  hinnan  resomces  files  of  potential  providers  to 
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important  to  customers  that  expect  many  of  their  displaced  employees  to  be 
hired  by  the  chosen  provider.  The  use  of  technology,  training  opportunities 
for  employees,  and  financial  stability  of  the  firm  are  good  predictors  of  the 
future  performance  of  the  provider.  Finally,  many  customers  are  concerned 
with  providers’  policies  on  social  issues  like  the  use  of  small,  minority-,  or 
women-owned  businesses  and  environmental  policies  such  as  energy 
efficiency  programs. 

Several  customers  used  performance  metrics  such  as  these  to  narrow 
the  competition  to  a  subset  of  providers  that  could  meet  all  of  their 
performance  and  cost  objectives  and  then  based  the  final  decision  on  other 
factors  that  are  not  easily  quantifiable.  One  such  factor  is  the  cultural  fit 
between  the  customer  and  provider  organizations.  A  good  cultural  fit 
between  the  customer  and  provider  facilitates  the  alignment  of  priorities  and 
communication  between  the  two  organizations.  (These  dimensions  of  the 
sourcing  process  are  discussed  below.)  Many  customers  judged  cultural  fit 
through  interviews  with  prospective  project  managers  for  the  provider 
organizations  and  through  site  visits  to  other  customers  of  those  providers. 

When  we  asked  firms  how  customers  ranked  these  evaluation  categories 
in  terms  of  importance,  we  received  varying  answers,  depending  on  the 
strategic  goals  that  the  customer  wanted  to  accomplish  through  outsourcing. 
One  provider  told  us  that  most  of  its  customers  are  primarily  concerned  with 
lowering  the  costs  associated  with  facility  management  services.  This  firm 
attributed  this  to  the  fact  that  the  level  of  services  its  organization  could 
provide  would  more  than  meet  customer  expectations.  At  the  other  extreme, 
one  customer  did  not  include  price  in  the  source  selection  decision.  It  based 
its  decision  primarily  on  technical  capability  and  training.  The  price  was 
negotiated  after  the  fact. 

Several  customers  told  us  that  they  used  performance  metrics  and 
weights  in  the  source  selection  process  to  help  make  the  process  as  objective 
as  possible.  Some  constructed  score  cards  with  metrics  and  associated 
weights  that  allowed  them  to  calculate  an  overall  score  for  each  of  the  top 
competing  providers.  One  large  firm  said  that  its  corporate  leaders  were 


identify  any  outstanding  liabilities  associated  with  environmental,  product  liability,  or 
worker  safety  issues. 
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especially  concerned  that  the  source  selection  process  be  objective,  because 
many  of  the  prospective  providers  were  also  clients.  The  corporate  leaders 
wanted  to  minimize  the  risk  of  those  providers  that  were  not  selected  taking 
their  business  elsewhere  because  of  perceived  biases  in  the  source  selection 
process. 

As  mentioned  earlier,  customers  may  use  different  metrics  to  provide 
information  during  the  various  stages  of  the  sourcing  process.  This  is 
illustrated  by  the  customer  firm  discussed  above  that  did  not  include  cost  in 
its  source  selection  criteria.  This  firm’s  primary  reason  for  outsourcing  its 
facility  management  services  was  to  reduce  its  expenses  because  it  faced  a 
dramatic  cut  in  the  facilities  budget  that  it  could  not  satisfy  within  the 
required  time  frame  with  its  in-house  organization.  However,  once  it 
determined  that  external  providers  could  maintain  services  at  the  reduced 
budget  level,  the  firm  chose  the  provider  that  best  met  its  performance  needs. 
The  quality  of  services  and  ability  of  the  provider  to  meet  future  needs 
entered  heavily  into  this  customer’s  selection  criteria  because  of  the  firm’s 
reliance  on  data  centers  and  the  importance  of  the  appearance  of  the  facilities 
to  clients  who  visited  its  offices. 
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Performance  Metrics  Help  Customers  and 
Providers  Manage  Their  Relationships 


Customers  and  providers  each  use  different  performance  metrics  in  a 
variety  of  ways  to  help  them  manage  their  relationships  with  one  another. 

Customers  use  performance  metrics  and  the  associated  goals  to 
communicate  their  objectives  and  expectations  to  their  providers.  The 
metrics  themselves  indicate  the  dimensions  of  performance  that  most  directly 
contribute  to  the  customer’s  strategic  goals,  and  the  goals  convey  information 
about  the  level  of  performance  that  the  customer  expects  in  those  areas.  For 
example,  a  customer  concerned  about  lost  employee  productivity  because  of 
facilities  problems  (e.g.,  bumed-out  light  bulbs)  may  track  cycle  time  for 
unscheduled  maintenance.  The  goal  for  this  metric  would  then  convey  the 
maximum  amount  of  time  that  is  acceptable  for  employees  to  wait  for  a 
service  request  to  be  satisfied.  Once  the  metrics  and  goals  are  in  place, 
customers  use  them  to  track  whether  their  providers  are  meeting  their 
expectations  and  needs.  In  the  next  chapters,  we  provide  details  about  how 
firms  choose  metrics  and  goals  and  about  their  processes  for  reviewing 
performance. 

Customers  sometimes  use  metrics  to  create  formal  monetary  incentives 
for  their  providers;  however,  we  observed  this  less  often  than  we  had 
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anticipated  at  the  beginning  of  the  study.  Instead,  the  primary  use  of  metrics 
seems  to  be  to  provide  information  that  the  customer  and  provider  use  to 
improve  performance  and  reduce  cost  over  time.  Several  customers  told  us 
that  they  hesitate  to  tie  metrics  to  award  fees  because  of  the  difficulty  of 
choosing  a  set  of  metrics  that  conveys  priorities  without  leading  to  distortions 
in  provider  performance.^  For  example,  too  much  emphasis  on  building 
maintenance  activities  may  lead  a  provider  to  forgo  some  grounds-keeping 
activities,  such  as  fertilizing  and  priming  shrubs,  that  its  customer  values.  It 
is  especially  difficult  to  choose  an  ideal  set  of  metrics  at  the  beginning  of  a 
long-term  relationship.  Given  these  challenges,  we  see  firms  selectively 
attaching  incentives  to  overarching  measures  such  as  cost  savings,  e.g., 
sharing  savings  above  a  threshold  level,  and  overall  customer  satisfaction, 
e.g.,  a  bonus  based  on  achieving  a  target  level  of  customer  satisfaction. 

These  metrics  are  likely  to  be  relevant  throughout  the  duration  of  a  contract. 
Those  customers  that  chose  not  to  tie  metrics  to  award  fees  used  other  tactics 
to  create  incentives  for  their  providers.  Many  providers  told  us  that  the 
possibility  of  a  customer  renewing  a  contract  without  rebidding  it  or 
expanding  scope  without  competing  the  additional  work  is  a  powerful 
incentive  to  meet  and  exceed  customer  expectations. 

Several  customers  noted  that  facility  management  services  are  often  at 
the  top  of  the  list  when  corporate  leadership  looks  for  areas  to  cut  costs. 
Customers  in  these  situations  use  performance  metrics  to  help  justify  the  cost 
of  providing  a  given  level  of  services.  One  customer  that  participates  in  a 
broad  benchmarking  exercise  uses  the  benchmarking  results  to  demonstrate 
to  its  corporate  leaders  that  its  costs  are  low  and  that  the  level  of  facilities 
services  will  have  to  drop  if  budgets  fall.  Similarly,  providers  use  metrics  and 


^  The  principal-agent  literature  formalizes  the  difficulties  associated  with  incentive 
contracts.  Holmstrom  and  Milgrom  (1991)  derive  optimal  contracts  for  agents  that  perform 
several  distinct  tasks  for  principals  or  have  a  single  task  with  multiple  dimensions.  They 
demonstrate  that  a  fixed  wage  contract  that  is  independent  of  measured  performance  of  a 
certain  activity  can  be  optimal  when  it  is  difficult  to  measure  performance  of  other  activities 
that  compete  for  the  agent’s  time  and  attention. 

^^However,  even  simple  measures  like  cost  savings  can  lead  to  distortions.  For 
example,  placing  too  much  emphasis  on  the  current  cost  of  providing  facilities  services  may 
lead  a  provider  to  forgo  preventive  maintenance  (thus  increasing  long-term  costs)  that  its 
customer  would  be  willing  to  pay  for. 
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benchmarking  results  to  justify  their  budgets  to  their  customers.^^  However, 
the  metrics  customers  and  providers  use  to  justify  budgets  may  be  very 
different.  The  metrics  providers  use  for  this  purpose  may  contain  much  more 
detail  about  operations  than  customers  convey  to  high-level  corporate 
leaders.  For  example,  a  customer  manager  may  use  benchmarking  results  on 
the  costs  of  centralized  facilities  call  centers  to  demonstrate  to  corporate 
leaders  that  its  cost  compares  favorably  with  similar  firms.  The  provider 
may  use  call  volume  and  hold-time  metrics  to  justify  its  call  center  staffing 
level  to  the  customer  project  manager. 

Customers  and  providers  also  use  metrics  to  realign  workscope  with  the 
budget  when  funds  are  unexpectedly  short  or  plentiful.  If  budgets  are  cut 
midyear,  metrics  help  both  customer  and  provider  project  managers  make 
informed  decisions  about  which  activities  can  be  scaled  back  or  eliminated, 
e.g.,  carpet  cleaning  or  painting.  Similarly,  metrics  help  identify  activities 
that  would  most  benefit  from  any  additional  funds,  e.g.,  cosmetic  services 
that  would  increase  customer  satisfaction  or  preventive  maintenance  that 
was  previously  eliminated  from  the  workscope. 

Finally,  providers  use  performance  metrics  to  demonstrate  how  they 
have  added  value  to  the  customer  organizations.  More  than  one  provider  said 
that  their  best  project  managers  are  attracted  to  those  customers  that 
effectively  use  performance  metrics  because  metrics  enhance  a  manager’s 
ability  to  understand  and  meet  or  exceed  his/her  customer’s  expectations. 


Although  nomenclature  differed  (e.g.,  fixed  price,  cost  plus,  or  time  and  materials), 
many  of  the  commercial  facility  management  contracts  can  be  characterized  as  follows.  A 
contract  generally  has  a  fixed  annual  budget  that  the  provider  cannot  exceed,  and  the 
provider  is  not  obligated  to  spend  the  entire  amount.  Expected  prices  for  materials  and  labor 
are  used  to  construct  the  budget,  and  actual  prices  determine  provider  expenditiares.  The 
provider  receives  a  fee  that  is  exogenous  to  expenditmes. 
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Metrics  Help  Providers  Continuously 
Improve  Their  Processes  and  Performance 


•  Provider  HQ  develops  key  performance 
measures  and  goals 

•  Site  managers  regularly  report  performance  in 
these  specified  areas  to  HQ 

•  HQ  benchmarks  performance  across  sites  and 
shares  results  with  site  managers 

•  HQ  identifies  best  practices  and  spreads  them  to 
other  sites 
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Performance  metrics  help  provider  firms  implement  a  process  to 
continuously  improve  performance  over  time.  Continuous  improvement  is  a 
never-ending  process  of  identifying  underperformance  of  a  service,  identifying 
and  fixing  the  root  causes  of  the  poor  performance,  and  then  looking  for  the 
next  opportunity.  Providers  use  metrics  to  identify  performance  constraints 
or  outliers,  “drill  down”  to  understand  how  to  improve  processes,  and  verify 
improvements  in  performance  over  time. 

This  chart  outlines  a  generic  continuous  improvement  process  that  we 
formed  based  on  a  combination  of  several  providers’  process  improvement 
practices.  First,  staff  in  the  corporate  headquarters  of  the  provider 
organization  work  with  site  managers  to  identify  the  important  dimensions  of 
providing  a  service  and  to  develop  performance  measures  and  associated 
goals  for  these  key  performance  drivers.  This  communicates  to  all  site 
managers  those  areas  of  performance  that  are  most  important  for 
demonstrating  value-added  and  satisfying  customer  needs.  Next,  site 
managers  report  at  specified  time  intervals  their  performance  in  these  areas. 
One  provider  goes  through  this  process  quarterly.  Third,  corporate 
headquarters  staff  compile  information  to  benchmark  performance  across 
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sites.  This  information  is  shared  with  all  site  managers.  Finally, 
headquarters  staff  use  metrics  to  investigate  variances  in  performance  across 
sites.  For  example,  if  at  one  site  the  provider  does  not  fulfill  new  orders  for 
services  as  quickly  or  accurately  as  it  does  at  others,  headquarters  may  check 
the  accuracy  of  the  data  systems  or  segment  times  for  specific  parts  of  the 
order  fulfillment  process.  When  variances  result  from  process  innovations, 
headquarters  staff  share  those  innovations  with  management  and  staff  at 
their  other  sites.  Because  of  the  difficulty  of  finding  “apples-to-apples” 
comparisons  in  benchmarking  exercises,  even  among  one  provider’s  customer 
sites,  the  last  step  is  crucial  to  the  success  of  a  continuous  improvement 
program. 

To  the  extent  that  problems  are  brought  to  light  while  “drilling  down”  to 
understand  variances,  metrics  help  site  managers  know  where  to  concentrate 
limited  training  resources.  In  some  cases,  corporate  headquarters  may  send 
a  team  to  help  a  site  manager  improve. 


15  Performance  is  also  compared  with  organizations  that  are  not  customers  of  the 
provider,  even  competitors. 
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Case  Study:  One  Provider’s  Use  of 
Benchmarking  to  Support  Continuous  Improvement 


*  Database  contains  over  80  metrics  that  reflect 
critical  success  factors 

.  Metrics  are  displayed  in  graphic  form  for  easy 
viewing 

•  Individual  site  manager  can  easily  examine  his/her 
site’s  performance  relative  to  performance  at  the 
provider’s  other  sites 

>  Outliers  stimulate  communication  across  sites  to 
understand  variances 

.  Headquarters  uses  a  formal  process  to  implement 
and  refine  changes 
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We  next  examine  more  carefully  one  particular  provider’s  continuous 
improvement  program.  This  commercial  firm  provides  third-party  facility 
management  services  to  many  commercial  buyers  at  many  sites.  The  buyer 
at  each  site  requests  a  different  bundle  of  services.  To  help  manage  these 
services  and  improve  them  over  time,  the  provider  maintains  a 
comprehensive  database  of  over  80  metrics  that  reflect  seven  dimensions  of 
performance  that  the  company  has  identified  as  critical  to  success  across  a 
range  of  services:  customer  service,  marketing  and  leasing,  maintenance  and 
engineering,  administration,  human  resources,  accounting  and  reporting,  and 
construction  management.  Some  metrics  measure  dimensions  of 
performance  directly  relevant  to  the  customer  at  each  site.  Others  measure 
key  processes  the  provider  uses  to  generate  this  performance.  Current  and 
historical  information  on  these  metrics  can  be  accessed  via  the  internet 
through  a  company  home  page  for  each  site.  The  metrics  are  displayed  in 
graphic  format  for  easy  viewing  and  are  normalized  for  comparison  across 
sites  where  appropriate. 

This  database,  which  is  updated  monthly,  supports  day-to-day 
management  at  each  site  and  longer-term  efforts  to  improve  performance. 
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On  a  day-to-day  basis,  the  database  is  a  source  of  information  to  calculate  the 
selected  metrics  that  the  customer  has  requested  for  regular  reporting.  The 
provider  can  also  compare  more-detailed  metrics  internally  to  verify  that  its 
processes  are  working  as  expected.  Corporate  headquarters  can  use  the 
database  to  compare  performance  at  different  sites  as  part  of  its  ongoing 
effort  to  compare  and  reward  its  managers  with  bonuses  or  future  career 
opportunities. 

Longer-term  improvement  can  flow  from  informal  and  formal  corporate 
efforts  to  change  performance.  Informally,  on  a  routine  basis,  a  site  manager 
can  easily  examine  performance  at  his/her  site  over  time  and  compare  that 
performance  with  other  company  sites.  Headquarters  staff  hope  that  easy 
access  to  cross-site  benchmarking  information  will  stimulate  communication 
across  sites  as  managers  seek  to  understand  variances  in  results  along 
specific  performance  dimensions.  Of  course,  not  all  metrics  are  relevant  to  all 
sites,  depending  on  the  services  provided  in  each  case. 

More  formally,  headquarters  can  introduce  a  new  practice  and  use  the 
database  to  measure  the  effects  of  the  change.  A  new  practice  might  be  a 
new  training  program,  a  centralized  call  center,  or  a  new  preventive 
maintenance  program.  This  provider  uses  a  formal  process  to  introduce  a 
change  at  one  or  more  sites.  The  first  step  in  this  provider’s  program  is  to 
design  a  practice  for  providing  a  given  service  and  an  execution  plan  for  its 
implementation.  The  second  step  is  to  train  its  employees  to  execute  the 
practice.  The  third  step  is  to  implement  the  practice  at  relevant  customers’ 
sites.  The  fourth  step  is  to  use  the  database  to  measure  performance  across 
sites.  The  fifth  step  is  to  audit  processes  at  sites  to  understand  variances  in 
performance.  Two  regions  are  randomly  chosen  for  third-party  site  audits 
each  year.  Audits  seek  greater  detail  than  the  database  routinely  reflects. 

The  detail  sought  is  tailored  to  the  variances  observed.  When  a  change  has 
been  made,  audits  carefully  track  implementation  to  verify  that  the  change  is 
proceeding  as  planned.  The  sixth  step  is  to  repeat  the  process  by  redesigning 
the  practice  as  needed  to  take  advantage  of  process  improvement. 


In  addition,  this  benchmarking  data  can  help  the  provider  bid  more  accurately  in 
competitions  for  new  work. 
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3.  CHOOSING  METRICS 
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The  next  three  chapters  address  the  metrics  that  customers  and 
providers  use  to  manage  their  partnership  relationships.  In  this  chapter,  we 
discuss  how  firms  choose  the  best  set  of  metrics.  We  examine  the  process 
that  customers  and  providers  go  through  to  choose  performance  metrics  that 
align  the  priorities  of  the  provider  with  those  of  the  customer,  the 
characteristics  of  these  metrics,  and  patterns  in  the  use  of  metrics.  Finally, 
we  offer  some  examples  of  metrics  that  we  observed  during  our  site  visits 
with  customers  and  providers  of  facility  management  services. 

The  focus  of  this  chapter  is  on  metrics  used  to  manage  the 
customer/provider  relationship,  but  many  of  the  principles  apply  to  metrics 
used  earlier  in  the  sourcing  process.!'^ 


Outline 


.  The  many  roles  of  performance  metrics  in  the 
sourcing  process 

.  Choosing  the  appropriate  set  of  metrics 
.  Collecting  data  and  generating  metrics 

•  Managing  performance  and  cost  against  goals 

•  Implications  for  the  Air  Force 
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1"^  Appendix  B  contains  examples  of  metrics  used  in  the  source  selection  decision  as 
well  as  metrics  used  to  manage  the  relationship. 
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Customer  and  Provider  Jointly  Choose 
Metrics  that  Reflect  Customer  Priorities 


Customers  tend  to  track  output-oriented  metrics 

Providers  may  track  additional  process-oriented 
metrics 

Metrics  may  be  either  qualitative  or  quantitative 
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Commercial  customers  and  providers  of  facility  management  services 
often  sit  down  together  to  choose  the  set  of  metrics  that  they  will  use  to 
govern  their  partnership.  A  customer  that  does  not  have  a  program  in  place 
to  measure  the  performance  of  its  in-house  provider  organization  or  that  is 
not  happy  with  its  internal  measures  may  begin  with  the  standard  group  of 
metrics  tracked  by  its  chosen  provider  and  then  add  and/or  subtract  metrics 
from  that  list  as  needed.  This  exercise  is  often  referred  to  as  the  alignment 
process  because  the  objective  is  to  choose  metrics  that  accurately  reflect  those 
dimensions  of  performance  and  cost  that  are  most  important  to  the  customer. 
Such  metrics  link  the  performance  outputs  of  facility  management  services 
with  both  the  short-run  and  long-run  strategic  corporate  outcome  goals  of  the 
customer.  18  Because  of  customers’  unique  goals  and  priorities,  metrics  differ 
across  customer/provider  relationships. 


18  Typical  overarching  outcome  metrics  of  the  customer  firm  include  profitability, 
meu-ket  share,  customer  satisfaction,  customer  retention,  employee  satisfaction,  employee 
retention,  and  employee  productivity.  In  contrast,  output  metrics  reflect  the  performance  of 
those  activities,  such  ais  facility  management  services,  that  enable  and/or  drive  the  core  firm 
outcomes.  Whereas  outcome  metrics  are  lagging  indicators  of  the  firm’s  performance,  output 
measures  are  leading  indicators.  Kaplan  £uid  Norton  (1996)  provide  a  detailed  discussion  of 
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During  our  interviews,  several  customers  discussed  the  link  between 
their  metrics  and  their  strategic  goals.  The  corporate  leadership  of  one 
customer  firm  was  especially  concerned  about  increasing  its  use  of  small 
businesses  and  minority-  and  women-owned  businesses  because  the  firm 
operates  in  a  regulated  industry.  To  promote  this,  the  customer  set  goals  for 
its  providers  to  do  a  certain  amount  of  business  with  these  firms  at  its  sites. 
The  providers  have  been  able  to  meet  and  exceed  these  goals  over  time. 

When  another  firm  with  an  extremely  large  campus  outsourced  the  provision 
of  its  food  services,  its  primary  concern  was  to  provide  appealing  lunch 
options  for  its  employees  to  keep  them  on-site  and  productive.  Thus,  the  food 
service  provider  was  evaluated  on  the  employee  participation  rate  and 
customer  satisfaction.  Another  customer  firm  with  facilities  spread  across 
several  states  has  regional  providers  that  manage  a  wide  range  of  services. 
Because  this  customer  was  concerned  that  its  regional  providers,  which  are 
competitors  in  other  situations,  might  not  partner  with  one  another  for  the 
good  of  their  customer,  the  customer  tracks  metrics  that  indicate  how  well  its 
providers  communicate  and  share  best  practices  among  themselves. 

During  our  interviews,  we  discerned  several  other  general 
characteristics  of  metrics  used  to  manage  the  customer/provider  relationship. 
Many  customers,  especially  those  that  have  transferred  a  significant  portion 
of  the  management  responsibility  to  their  providers,  prefer  to  track  output- 
oriented  metrics  rather  than  the  details  of  provider  processes.  One  customer 
firm  told  us  that  it  set  up  partnerships  with  world-class  providers  and  then 
adopted  a  “hands-ofF’  policy.  It  closely  tracks  the  level  of  performance  of  the 
services  provided,  but  it  does  not  worry  about  the  processes  the  providers  use 
to  meet  performance  goals.  For  example,  it  tracks  the  performance  of 
scheduled  preventive  maintenance  but  not  how  often  provider  technicians 
visit  each  building.  However,  some  customers  are  not  comfortable  relying 
only  on  output-oriented  metrics.  These  customers  expressed  concerns  about 
their  ability  to  define  and  measure  outputs  for  all  activities.  This  difficulty 
leads  them  to  use  a  mixture  of  output-  and  process-oriented  metrics.  One 
high-technology  firm  uses  both  t3^es  of  metrics  for  food  services,  tracking 

the  different  categories  of  metrics.  The  interested  reader  shotxld  note  that  they  use  the  term 
performance  measures  to  describe  those  that  we  refer  to  as  output  metrics. 
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fulfillment  of  specific  menu  guidelines  (e.g.,  number  of  entree  choices  offered) 
in  addition  to  customer  satisfaction. 

Providers  use  many  process-oriented  metrics  in  addition  to  the  output 
metrics  that  their  customers  track.  These  process-oriented  metrics  help 
diagnose  the  underlying  causes  of  poor  performance  and  are  an  integral  part 
of  continuous  improvement  programs,  For  example,  slow  cycle  times  for 
work  orders  could  arise  from  any  number  of  problems  such  as  high  absentee 
rates  for  technicians,  poor  inventory  management  practices  for  parts,  or  an 
inefficient  reporting  system.  Process-oriented  metrics  help  providers  isolate 
problems  and  target  resources  to  correct  them. 

Finally,  customers  and  providers  commonly  track  both  qualitative  and 
quantitative  metrics.  The  most  common  example  of  qualitative  metrics  are 
those  that  track  customer  satisfaction,  such  as  overall  satisfaction  with 
services  provided  or  satisfaction  with  the  execution  of  a  particular  work 
order,  e.g.,  replacing  missing  ceiling  tiles.  An  example  of  a  quantitative 
metric  is  the  percentage  of  preventive  maintenance  activities  that  are 
completed  on  schedule. 


Kaplan  and  Norton  (1996)  refer  to  these  process-oriented  metrics  as  diagnostic 
measxu-es. 


During  our  site  visits,  we  discerned  that  there  are  several  patterns  in 
the  implementation  of  performance  metrics  used  to  manage  the  relationship 
between  customers  and  providers  of  facility  management  services.  First, 
customers  tend  to  focus  on  a  few  key  metrics  for  each  service  provided  in 
order  to  easily  convey  to  providers  those  dimensions  of  performance  that  are 
most  important.  20  For  example,  virtually  every  high-technology  customer 
that  we  interviewed  told  us  that  it  measures  the  cost  of  employee  moves. 
This  is  an  important  dimension  of  facilities  performance  for  high-technology 
firms;  these  firms  move  their  professional  staff  frequently  as  project  teams 
repeatedly  form  and  disband  because  of  the  short-lived  nature  of  most 
projects.  One  provider  told  us  that  its  customers  typically  specify  two  to  four 
metrics  per  activity  or  service.  However,  the  customer  must  be  careful  to 
ensure  that  the  set  of  metrics  for  any  service  or  group  of  services  is  large 
enough  to  cover  the  entire  range  of  important  performance  dimensions. 

As  part  of  the  process  of  aligning  customer  and  provider  priorities,  the 
set  of  metrics  used  by  a  given  customer  and  provider  is  refined  throughout 

20However,  one  customer  we  interviewed  chose  to  measure  virtually  every  aspect  of 
cost  and  performance. 
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the  relationship  lifetime.  Metrics  evolve  over  time  as  the  provider  and 
customer  learn  more  (from  experiences  both  inside  and  outside  of  the 
partnership)  about  which  aspects  of  performance  contribute  most  to 
satisfying  the  goals  of  the  customer  or  as  the  business  needs  of  the  customer 
change. 21  This  evolution  is  especially  important  for  measuring  those  services 
that  are  difficult  to  specify,  such  as  landscaping.  One  customer  told  us  that  it 
has  refined  its  performance  measures  for  landscaping  services  to  reflect  the 
importance  of  appearance  across  different  outdoor  “zones.”  Another  customer 
told  us  that  it  examines  its  metrics  over  time  to  make  sure  that  each 
continues  to  add  value.  It  eliminates  those  metrics  that  no  longer  add  value 
and  adds  new  ones  as  priorities  shift.  As  will  be  discussed  below,  ongoing 
communication  between  a  customer  and  its  provider  creates  opportunities  to 
refine  and  update  the  set  of  metrics  used  to  manage  the  relationship. 

Finally,  to  create  the  flexibility  to  adjust  metrics  over  time,  many 
customers  separate  the  specific  metrics  that  they  will  use  to  manage  their 
partnerships  from  their  contracts  with  providers.  In  place  of  specific  metrics, 
one  provider  recommends  inserting  a  clause  in  the  contract  that  indicates 
that  the  customer  and  provider  will  reach  an  agreement  about  a  set  of 
metrics  to  track  over  time  and  how  those  metrics  will  be  reported  to  the 
customer.  However,  some  customers  do  specify  an  initial  set  of  metrics  in  the 
contract  and  then  either  modify  the  contract  to  adjust  those  metrics  or 
informally  change  them  over  time  outside  of  the  original  contract.22  When  a 
customer  does  leave  metrics  out  of  the  contract,  it  reaches  agreement  about 
initial  metrics  with  its  provider  soon  after  the  contract  is  signed. 

There  is  one  circumstance  in  which  the  customers  and  providers  we 
interviewed  unanimously  elected  to  formally  specify  metrics  in  the  contract: 
when  those  metrics  were  directly  tied  to  provider  compensation.  However,  as 
noted  earlier,  many  customers  and  providers  avoid  formalizing  incentives 
with  performance  metrics  because  of  the  difficulty  of  coming  up  with  the 
perfect  set  of  metrics  that  accurately  reflects  all  the  important  dimensions  of 
performance  at  the  very  beginning  of  their  relationship.  One  provider  said 
that  it  had  to  dismiss  several  project  managers  who  performed  well  according 

21  See  also  Courty  and  Marschke  (1997). 

22  Some  customers  have  hybrid  approaches  in  which  some  metrics  are  included  in  the 
contract  and  others  are  determined  outside  of  the  contract. 
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to  metrics  but  ignored  other  aspects  of  performance  that  their  customers 
valued  (which  led  to  low  customer  satisfaction  ratings). 

The  next  three  charts  provide  examples  of  metrics  that  many  of  the 
customers  and  providers  we  interviewed  use  to  manage  their  relationships. 


Many  Customers  and  Providers  Track 
Cost  and  Customer  Satisfaction 


•  Financial  metrics 

-  budget  vs.  actual  expenditure 
-cost  of  performing  an  activity 

•  Customer  satisfaction 

-measure  timeliness,  quality,  and  overall 
satisfaction 

-target  direct  users  of  services,  managers, 
executives 


P 

I 
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Although  the  specific  metrics  vary  across  customer/provider 
relationships,  several  topic  areas  were  common  to  many  of  the  firms  that  we 
interviewed.  Two  kinds  of  metrics  that  we  observed  frequently  are  financial 
metrics  and  customer  satisfaction  metrics. 

Common  financial  metrics  include  those  that  track  the  actual 
expenditures  of  the  provider  against  the  facilities  budget  and  the  cost  of 
performing  a  given  activity.  Because  the  demands  for  facilities  services  can 
be  difficult  to  predict,  at  least  one  customer  that  we  interviewed  said  that  it 
tracks  expenditures  against  the  budget  monthly  so  that  the  scope  of  work  can 
be  adjusted  up  or  down  as  needed  to  stay  within  the  budget.  Several 
customers  said  that  they  try  to  take  advantage  of  budget  underruns  by 
adding  out-of-scope  work  or  expanded  services  such  as  additional  preventive 
maintenance.  Customers  and  providers  also  track  the  cost  of  performing  key 
activities  as  part  of  benchmarking  exercises.  Building  maintenance  and 
utility  costs  per  square  foot  are  commonly  measured  across  sites  for  such 
studies. 

Innovative  customers  and  providers  have  constructed  many  different 
measures  of  customer  satisfaction.  Customer  satisfaction  is  measured  for 
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different  groupings  of  services,  from  the  overall  package  of  services  provided 
down  to  specific  services  such  as  cleaning  break  rooms.  Also,  firms  measure 
many  dimensions  of  performance;  a  few  that  were  mentioned  often  in  our 
interviews  are  overall  performance,  timeliness  of  services,  quality,  manners 
of  the  technicians,  and  responsiveness  to  change.  Finally,  satisfaction  may  be 
measured  at  multiple  levels  in  customer  firms,  from  the  direct  users  of 
services  such  as  the  staff  member  who  requests  a  new  light  bulb,  to  the 
project  manager  who  oversees  the  provider,  to  the  high-level  leaders  in  the 
firm. 
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Quantitative  Measures  of  Performance 
Are  Also  Common 


•  Work  order  cycle  time,  percentage  up-time  for 
security  and  computer  systems 

•  Cycle  time  for  preventive  maintenance  actions, 
percentage  of  preventive  maintenance  actions 
completed  on  time 

•  Number  of  worker’s  compensation  claims,  lost 
work  days  from  on-the-job  accidents,  number  of 
safety  incidents 
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In  addition  to  financial  metrics  and  qualitative  customer  satisfaction 
metrics,  all  of  the  customers  and  providers  we  talked  to  track  quantitative 
metrics  that  target  performance  of  specific  services.  Examples  of  output- 
oriented  metrics  are  the  cycle  times  to  complete  work  orders  and  the 
percentage  of  time  security  systems  are  operational.  One  customer  tracks  the 
percentage  of  up-time  for  the  computers  at  its  data  centers.  Providers  also 
track  process-oriented  metrics  like  the  percentage  of  preventive  maintenance 
tasks  that  are  completed  on  schedule  and  cycle  times  for  preventive 
maintenance  tasks. 

Many  customers  and  providers  emphasize  their  commitment  to  safe 
work  environments  by  tracking  safety-oriented  metrics.  Frequently  observed 
examples  include  the  number  of  worker’s  compensation  claims,  the  number  of 
lost  work  days  from  on-the-job  accidents,  and  the  number  of  safety-related 
incidents  during  a  specified  time  period. 
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In  addition  to  metrics  that  directly  target  the  cost  and  performance 
associated  with  the  provision  of  facility  management  services,  a  subset  of 
customers  is  interested  in  monitoring  how  well  their  providers  address 
human  resources  (HR)  issues  and  special-interest  social  issues.  Customer 
interest  in  tracking  these  kinds  of  metrics  appears  to  stem  both  from  the 
effect  human  resources  policies  and  practices  have  on  the  provision  of 
services  and  from  an  interest  in  forming  a  partnership  with  a  provider  whose 
culture  closely  matches  its  own. 

Many  customers  are  interested  in  tracking  employee  turnover  because 
they  believe  the  performance  of  services  that  involve  higher  levels  of  skills, 
institutional  knowledge,  or  personal  interaction  with  customer  employees 
benefits  from  stable  workforces.23  Customers  also  track  providers’ 
implementation  of  human  resources  policies  through  metrics  such  as  the 
number  of  equal  employment  opportunity  suits  filed  against  the  provider 
organization. 


Some  Firms  Pay  Special  Attention  to 

Employee  and  Social  issues 


•  Human  resources 

-employee  turnover 

-equal  employment  opportunity  suits 

•  Special  interest 

-amount  of  business  done  with  small,  minority- 
owned,  and  women-owned  businesses 
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stable  workforce  also  helps  create  a  more  safe  and  secure  working  environment. 
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One  provider  told  us  that  special-interest  metrics  are  tracked  less  often 
than  other  kinds  of  metrics;  however,  we  observed  that  many  customers  are 
interested  in  the  amount  of  business  their  providers  do  with  small,  minority- 
owned,  and  women-owned  businesses,  either  at  a  particular  site  or 
companywide.  This  is  especially  true  for  those  customers  that  do  business 
with  the  government  or  are  in  regulated  industries  such  as  banking. 
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4.  COLLECTING  DATA  AND  GENERATING  METRICS 


.  The  many  roles  of  performance  metrics  in  the 
sourcing  process 

.  Choosing  the  appropriate  set  of  metrics 
Collecting  data  and  generating  metrics 

•  Managing  performance  and  cost  against  goals 

•  Implications  for  the  Air  Force 
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In  this  chapter,  we  discuss  how  customers  and  providers  collect  data  to 
generate  metrics.  Again,  we  examine  those  metrics  used  to  manage  the 
relationship  between  a  customer  and  its  provider. 
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Firms  Recommend  Specifying  in  Advance 

Methods  for  Collecting  Data  and  Generating  Metrics 


Methods  include 
-customer  surveys 
-  management  information  systems 
-audits 

Reliability  of  data  can  be  a  concern 


Cost  <  Benefits 
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Choosing  the  set  of  metrics  that  will  be  used  to  manage  the 
customer/provider  relationship  takes  careful  consideration.  Customers  and 
providers  emphasized  the  importance  of  understanding  how  the  data  to 
support  potential  metrics  would  be  collected  and  how  the  metrics  would  be 
generated.  This  avoids  wasted  time  arising  from  unrealistic  expectations 
about  the  use  of  metrics  that  might  lead  to  disputes. 

There  are  several  ways  in  which  customers  and  providers  collect  data  to 
generate  the  metrics  that  they  track,  including  customer  surveys, 
management  information  systems,  and  audits.  The  next  several  charts 
discuss  each  of  these  in  detail. 

Virtually  every  firm  that  we  interviewed  said  that  it  has  trouble 
collecting  accurate  data  necessary  for  the  generation  of  useful  metrics.^s 

^^Although  specific  metrics  to  manage  the  relationship  may  not  be  chosen  until  after 
the  contract  has  been  signed,  the  customer  and  provider  should  have  a  fairly  good  idea  about 
the  general  categories  of  metrics  that  will  be  used.  Based  on  this,  the  firms  can  discuss 
methods  for  collecting  data  and  generating  metrics  early  in  their  relationship. 

25Many  facility  services  contracts  are  variants  of  time  and  materials  contracts.  Thus, 
providers  indicated  that  they  are  concerned  about  accuracy  of  cost  data,  such  as  the  number 
of  hours  provider  employees  spend  working  at  a  customer  site  and  materieils  used  at  a 
customer  site,  for  contract  reimbursement,  in  addition  to  generation  of  useful  metrics. 
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However,  firms  have  increased  the  accuracy  of  their  data  through  such 
methods  as  automation  of  the  data  collection  process.  As  will  be  discussed 
below,  the  use  of  bar-code  scanners  virtually  eliminates  the  need  for 
technicians  to  enter  data  by  hand.  One  provider  added  a  manager  at  one  of 
its  best  customer’s  site  to  help  improve  data  entry.  This  person  reinforced 
employee  responsibility  for  data  accuracy  by  tracking  errors  back  to 
individuals  and  reporting  those  errors  to  their  supervisors.  This  provider 
emphasized  the  key  role  that  communication  plays  in  improving  data 
accuracy.  Not  only  is  it  important  to  track  errors  and  hold  people 
accountable  for  them,  it  also  is  important  to  communicate  to  employees  how 
others  (whom  they  may  never  see)  use  the  data  that  they  enter. 

As  customers  devolve  more  and  more  of  the  management  of  services  to 
external  providers,  they  rely  more  heavily  on  output  information  furnished  by 
those  providers.  Several  firms  we  spoke  with  indicated  that  they  took  steps 
to  satisfy  themselves  that  they  can  trust  metrics  generated  by  their 
providers.  As  will  be  discussed  below,  customers  spot-check  providers’  work 
and  perform  their  own  customer-satisfaction  surveys.  They  also  periodically 
audit  their  providers’  data  collection  and  metrics  generation  processes. 26 

Finally,  firms  emphasized  that  the  cost  of  generating  accurate  metrics 
should  never  outweigh  the  benefits  associated  with  those  metrics.  The 
generation  and  examination  of  metrics  are  costly  to  both  customers  and 
providers,  and  firms  try  to  eliminate  those  metrics  that  do  not  add  value 
above  the  associated  costs.  We  return  to  this  point  in  the  next  chapter. 


26  Customers  may  perform  the  audits  themselves  or  hire  consultants. 
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||  Surveys  Are  Used  to  Collect  Customer  || 

Satisfaction  Data 

•  initiated  by  both  customers  and  providers 

I 

•  Directed  to  both  customer  managers  and  those 

that  directly  benefit  from  services 

% 

•  Hardcopy,  email,  or  phone  calls 

-  at  the  time  of  service,  random  sampies 

periodically,  yearly 

% 

4, 

% 

•  Simple  format 

1 

Too  frequent  surveys  can  lead  to  low 

response  rates 

L _ i 

21  ^ 

We  observed  heterogeneity  in  the  processes  firms  use  to  conduct 
customer  satisfaction  surveys.  This  heterogeneity  is  an  indication  that 
innovative  firms  are  still  experimenting,  trying  to  determine  the  best 
methods  to  generate  the  information  they  need  to  evaluate  and  improve 
performance. 

Both  customers  and  providers  conduct  customer-satisfaction  surveys. 
Providers  use  the  surveys  both  to  satisfy  customer  requests  for  information 
and  to  track  the  performance  of  the  site  team,  of  individual  staff,  and  of  their 
subcontractors.  Customers  conduct  their  own  surveys  to  verify  the  level  of 
performance  reported  by  their  providers.  These  surveys  target  personnel  at 
several  levels  of  the  customer  organization — ^the  end  users  of  the  services 
such  as  the  occupants  of  a  building  whose  heating  and  cooling  systems  are 
maintained  by  an  external  provider,  the  project  manager  who  maintains  the 
relationship  with  the  provider,  and  corporate  leaders. 

Customers  and  providers  collect  information  through  hardcopy,  email, 
or  phone  surveys,  or  through  some  combination  of  these.  Firms  time  their 

None  of  the  firms  we  interviewed  had  outsourced  their  customer-satisfaction 

surveys. 
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surveys  to  target  different  groups.  Some  firms  ask  the  technicians  to  leave 
survey  cards  with  the  end  user  each  time  a  service  request  is  completed  or 
send  email  shortly  thereafter.  Others  send  surveys  several  times  a  year  to 
random  groups  of  customer  employees.  One  provider  surveys  customer 
managers  yearly.  Customers  and  providers  tend  to  agree  that  effective 
surveys  have  simple  formats  and  easy-to-answer  questions. 

Many  customers  and  providers  seem  to  be  aware  of  the  dangers  of 
conducting  excessive  surveys.  Several  firms  told  us  that  they  redesigned 
their  survey  strategies  after  experiencing  low  response  rates  associated  with 
surve3dng  customer  employees  too  frequently.  However,  other  firms  seem  to 
be  unconcerned  about  this  issue.  One  customer/provider  group  that  we 
talked  with  indicated  that  each  firm  surveys  employees  in  multiple  parts  of 
the  customer’s  organization  frequently.  It  even  appeared  that  these  firms 
have  overlapping  surveys. 
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ivlanagement  Information  Systems  Allow 
Simple,  Direct  Observation  of  Operations 


•  Systsms  automatically  calculate  metrics  of  interest 
to  customer  and  provider 

•  Providers  can  make  MIS  investments  that  are  not 
practical  for  customers 

•  Systems  are  often  based  on  commercial  off-the- 
shelf  software 

•  MIS  can  be  customized  for  individual  clients 

•  Customers  may  retain  their  data  if  contract  is 
terminated 
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As  discussed  above,  leading  providers  of  facility  management  services 
use  management  information  systems  (MIS)  to  automate  the  collection  of 
data  and  the  generation  of  operational  metrics  that  are  of  interest  to 
providers  and  their  customers.  Customers  use  this  information  to  track 
provider  performance,  and  providers  use  the  information  to  improve 
processes  and  control  costs. 

Customers  generally  do  not  have  internal  MIS  capabilities  that  are  as 
sophisticated  as  those  of  external  providers  because  of  the  large  investments 
required  to  sustain  these  systems.  A  customer  remarked  that  one  of  its 
providers  invests  $5-$10  million  each  year  in  its  MIS.  Customers  cannot 
afford  to  make  such  investments  in  tools  that  support  noncore  activities, 
whereas  a  single  provider  can  leverage  an  investment  in  its  MIS  across 
multiple  customers. 

Provider  MIS  tend  to  be  based  on  commercially  available  software, 
although  these  systems  vary  across  providers.  Some  systems  are  more 
powerful  and  complex  than  others,  and  some  have  been  customized.  In  fact, 
MIS  capability  is  one  of  the  primary  ways  facility  management  providers 
differentiate  themselves  in  the  market. 
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Some  providers  can  tailor  their  MIS  to  the  particular  needs  of  their 
customers.  For  example,  the  metrics  database  described  above  in  the  case 
study  can  be  expanded  to  include  metrics  requested  by  that  provider’s 
customers.  Even  though  the  systems  themselves  generally  are  proprietary, 
customers  can  build  into  their  contracts  the  right  to  retain  their  own  data, 

should  the  relationship  end  for  any  reason. 

The  next  two  charts  discuss  two  types  of  MIS  that  are  used  by  several 

large,  world-class  providers. 
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Many  large  customers  use  centralized  call  centers  to  manage  internal 
requests  for  facilities  services. Often,  a  customer’s  call  center  is  operated 
by  one  or  more  of  its  provider  organizations,  although  some  firms  have 
chosen  to  keep  this  function  in-house. ^9 

The  concept  behind  the  centralized  call  center  is  as  follows.  A  customer 
employee  calls  a  single  phone  number  to  report  a  facilities  problem  such  as  a 
clogged  drain  in  a  restroom  or  a  pest  control  need;  the  call  center  staff  records 
the  problem,  tasks  the  appropriate  service  provider  (either  internal  or 
external),  and  tracks  the  work  order  to  completion.  A  single  database  records 
this  information,  including  time  stamps,  the  affected  equipment,  the  cause  of 
the  problem,  materials  used,  and  the  services  provided. 

Customers  and  providers  use  call  center  databases  to  track  the 
following  kinds  of  metrics:  the  volume  of  calls,  response  times  to  those  calls, 
cycle  times  for  work  orders,  the  distribution  of  problems  across  locations, 

28Several  large  providers  mentioned  plans  to  implement  additional  call  centers  that 
can  be  shared  by  several  small  customers. 

^^Presentation  on  “Outsourcing  the  Facilities  Help  Desk  at  the  World  Bank — Panel 
Discussion,”  International  Facility  Management  Association  (IFMA),  Dallas,  October  1997. 
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frequency  of  problems  associated  with  specific  pieces  of  equipment,  and  the 
cost  associated  with  the  problems  observed. 

One  customer  has  a  call  center  for  each  of  the  three  regions  of  the 
country  in  which  it  has  offices.  These  centers  handle  all  requests  for  services 
that  cost  less  than  $500.  Together,  the  three  call  centers  receive  a  total  of 
225,000  calls  per  year.  The  associated  databases  allow  the  customer  and 
providers  to  manage  both  performance  and  costs.  For  example,  they  use  the 
information  generated  about  repeat  repairs  for  individual  pieces  of 
equipment  to  make  repair-vs. -replace  decisions  and  to  fine-tune  the 
preventive  maintenance  program.  This  customer  believes  that  its  call  centers 
have  resulted  in  higher-quality  services  at  lower  costs. 


-46- 


MIS  Example:  Bar-Code  Scanners 


•  Time  stamps  created  when  technician  scans  bar 
code  on  work  order,  equipment,  etc. 

•  Scanner  transfers  information  to  a  computer  at  the 

end  of  the  day  | 

•  information  helps  | 

-track  frequency  of  repair,  preventive 
maintenance  | 

-track  repair,  preventive  maintenance  cycle  | 

times  I 

if 

-create  preventive  maintenance  programs 

% 
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Many  large  providers  are  implementing  bar-code  scanner  technology  at 
the  sites  of  their  largest  customers.  These  scanners  essentially  eliminate  the 
need  for  field  technicians  to  record  information  by  hand.  Instead,  a 
technician  scans  a  work  order,  creating  a  time  stamp  marking  initiation  of 
work.  The  technician  also  scans  the  piece  of  equipment  to  be  serviced, 
recording  the  serial  number  and  type  of  equipment.  Parts  are  scanned  to 
record  materials  used  to  perform  the  work.  After  completing  the  service,  the 
technician  scans  the  work  order  again  to  create  a  closing  time  stamp.  This 
information  is  then  either  transmitted  or  downloaded  into  a  computer  at  the 
end  of  the  day.  The  technician  no  longer  needs  to  manually  keep  track  of 
his/her  time  or  the  materials  that  he/she  uses  during  the  day.  Providers 
indicate  that  this  technology  has  increased  data  accuracy  immensely. 

Providers  use  this  information  to  track  the  frequency  of  repair  of 
specific  pieces  of  equipment,  whether  preventive  maintenance  is  taking  place 
according  to  schedule,  and  the  cycle  times  for  repair  and  preventive 
maintenance.  This  information  can  then  be  used  for  purposes  such  as 
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designing  preventive  maintenance  schedules  for  the  customer  and 
determining  a  budget  for  the  work.^o 


sOMore  than  one  firm  expressed  some  doubt  that  the  benefits  of  bar-code  scanner 
technology  outweigh  the  high  costs.  One  provider  told  us  that  the  cost  of  each  bar-code 
scanner  (wand)  is  $790.  This  provider  supplies  each  technician  per  shift  with  a  s^ner.  ihe 
link  between  the  scanner  and  a  computer  is  $122  per  scanner,  $30  for  the  cable.  TOe 
complete  computerized  work  order  and  preventive  maintenance  software  system  is  $1990  tor 
the  first  user  and  $950  for  each  additional  user.  In  addition,  all  customer  equipment  would 
have  to  be  equipped  with  a  bar-code,  plus  all  equipment  would  have  to  be  entered  into  a 
master  database  for  reference.  This  is  still  a  somewhat  controversial  application  of 
technology  in  the  facility  management  industry.  Nonetheless,  we  expect  the  cost  of  this 
technology  to  fall  over  time,  making  it  more  cost-effective. 
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Audits  Supplement  Routine  Collection  of 
Performance  Data 


•  Customer  manager  may  spot-check  provider  to 
verify  reported  performance 

•  Provider  may 

-spot-check  self-performed  work 

-  perform  operational  and  financial  audits  of 
subs 

.  Third-party  auditors  provide  independent 
assessments  of  performance,  finances 

.  ISO  9000  ideas  are  common  but  not  always 
formally  implemented 


P 

I 


RAND  Project  AIR  FORCeI 


Customers  and  providers  of  facility  management  services  often  collect 
data  through  visual,  operational,  and  financial  audits  to  use  in  managing 
their  relationships.  These  audits  are  performed  by  the  customer,  provider,  a 
third-party  organization,  or  some  combination  of  these. 

First,  customer  managers  may  spot-check  the  performance  of  their 
provider  to  verify  that  the  services  provided  are  of  the  quality  reported  in 
their  metrics.  For  example,  one  customer  walks  the  grounds  periodically  to 
check  the  quality  of  the  landscaping  work.  As  discussed  above,  such  checks 
are  especially  useful  when  the  provider  is  in  charge  of  collecting  the  data  that 
is  used  to  generate  the  performance  metrics. 

Next,  managers  of  the  provider  organization  may  do  similar  spot-checks 
of  work  that  is  self-performed  or  subcontracted.  Providers  may  also  perform 
operational  and  financial  audits  on  their  subcontractors. One  provider  that 
self-performs  most  activities  but  contracts  out  some  services  such  as  custodial 

^^Operational  audits  are  similar  to  financial  audits.  An  operational  audit  may  consist 
of  measuring  the  quality  of  the  output  of  processes  or  sampling  processes  to  make  sure  that 
the  actual  implementation  of  processes  matches  company  policy. 
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work,  performs  operational  audits  on  its  custodial  subcontractor.  This 
provider  has  a  formal  process  for  performing  these  audits. 

Finally,  both  customers  and  providers  hire  third-party  auditors  to 
provide  independent  assessments  of  performance  and  financial  status.  This 
same  provider  employs  two  separate  auditors  to  evaluate  the  provision  of  food 
services  by  its  subcontractor.  The  provider  told  us  it  feels  more  comfortable 
having  multiple  assessments  of  performance  in  such  an  important  service 
area.  Another  provider  that  subcontracts  out  most  services  hires  a  third- 
party  auditor  to  perform  surprise  audits  at  its  sites.  One  customer  hires  one 
of  the  large  accounting  firms  to  perform  financial  audits  of  its  providers  to 
verify  cost  information  and  financial  practices.  It  uses  a  separate  auditor  to 
perform  operational  audits  of  its  providers.  This  customer  is  faced  with 
rapidly  decreasing  facilities  budgets,  and  it  uses  operational  audits  to  verify 
the  condition  of  its  facilities  to  make  sure  that  its  providers  are  not  forgoing 
valuable  preventive  maintenance  in  order  to  stay  within  stringent  budget 
targets. 

We  were  surprised  that  many  world-class  providers  of  facility 
management  services  are  not  ISO  9000  certified.32  when  we  asked  why,  we 
were  told  that  ISO  9000  ideas  are  valuable,  but  that  the  process  is  so 
cumbersome  and  costly  that  they  have  decided  not  to  implement  it  formally 
unless  customers  request  it.  Instead,  many  firms  go  through  what  appears  to 
be  a  self-certification  process  to  get  the  benefits  of  certification  without  the 
bureaucratic  burden.  Other  more  aggressive  firms  indicated  that  ISO  9000 
ideas  are  primitive  compared  with  their  internal  process  documentation  and 
audit  program. 


22The  objective  of  the  International  Organization  for  Standardization  (ISO)  is  to 
develop  generic  standards  to  promote  the  development  of  manufacturing,  communication, 
and  trade  among  business  operations.  ISO  9000  is  a  series  of  standards  that  define  a  basic 
quality  management  system  that  can  be  used  in  any  industry,  of  any  size,  anywhere  in  the 
world  (source:  www.Dpmfg.com/iso9000.html).  See  Johnson  (1993)  for  a  discussion  of  ISO 

9000  certification. 
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5.  MANAGING  PERFORMANCE  AND  COST  AGAINST  GOALS 


The  many  roles  of  performance  metrics  in  the 
sourcing  process 

Choosing  the  appropriate  set  of  metrics 
Collecting  data  and  generating  metrics 
Managing  performance  and  cost  against  goals 
Implications  for  the  Air  Force 
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In  this  chapter,  we  discuss  how  customers  and  providers  choose 
performance  and  cost  goals  and  how  they  manage  the  provision  of  facility 
management  services  against  those  goals. 
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Contract  may  specify  degree  of  improvement  over 
time  for  unusual  facilities 

Customer  corporate  leadership  may  impose 
cost/performance  goals 
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Innovative  customers  and  providers  combine  the  needs  and  desires  of 
the  customer  with  the  expertise  of  the  provider  to  derive  a  set  of  performance 
and  cost  goals  for  the  metrics  they  use  to  manage  their  relationship.  Rather 
than  representing  minimum  acceptable  performance  outputs,  these  goals 
represent  desired  levels  of  performance  and  cost.  Some  firms  take  the  goal¬ 
setting  process  a  step  further  by  differentiating  between  short-run  and  long- 
run  goals.  As  the  term  suggests,  short-run  goals  represent  outputs  the 
customer  desires  based  on  the  current  capabilities  of  the  provider  or 
improvements  that  could  be  made  within  the  immediate  measurement 
period.  In  contrast,  long-run  goals  convey  desired  significant  improvements 
in  provider  capabilities  that  are  possible  through  continuous  improvement 
programs.  Firms  tend  to  devote  much  of  their  time  and  resources  to  meeting 
long-run  goals. 

Both  customers  and  providers  noted  the  desirability  of  disassociating 
the  goal-setting  process  fi"om  engineering  standards  for  facilities  activities 
because  use  of  such  standards  as  goals  works  against  the  concept  of 
continuous  improvement.  Engineering  standards,  such  as  the  number  of 
technicians  needed  per  10,000  square  feet  of  office  space,  are  based  on  the 
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performance  that  can  be  expected  given  existing  processes  for  accomplishing 
activities.  Goals  that  represent  the  status  quo,  such  as  the  labor  cost 
associated  with  the  “standard”  number  of  technicians  needed  to  service 
10,000  square  feet,  do  not  provide  motivation  for  the  kinds  of  improvements 
that  customers  seek  and  that  leading  providers  use  to  differentiate 
themselves  in  the  market. 

Instead,  many  customers  and  providers  look  to  benchmarking  studies  to 
provide  information  about  feasible  improvements  in  performance  and  cost. 
Because  of  the  difficulty  of  finding  “apples-to-apples”  comparisons  of  facilities 
and  levels  of  services,  firms  focus  on  variances  in  performance  and  cost  and 
seek  to  understand  the  sources  of  those  variances. ^3  To  the  extent  that  top 
performance  in  an  activity  results  from  process  improvements  or 
technological  innovations  rather  than  differences  in  facilities’  characteristics 
or  uses  or  the  scope  of  services,  the  leading  level  of  performance  reflects  a 
feasible  goal  to  strive  for. 

Customers  with  facilities  that  have  unusual  uses  or  characteristics 
(such  as  specialized  laboratories)  will  have  difficulty  comparing  their  own 
performance  and  cost  with  the  results  of  benchmarking  studies.  In  these 
cases,  customers  and  providers  may  benefit  from  specifying  performance  and 
cost  goals  in  terms  of  percentage  improvements,  relative  to  current  levels, 
over  time. 

Finally,  in  some  organizations,  aggressive  internal  requirements  set  by 
corporate  leaders  drive  the  choice  of  facilities  goals.  As  discussed  above,  one 
customer’s  primary  motivation  for  outsourcing  its  facility  management 
services  was  an  upcoming  cut  in  its  facilities  budget.  The  performance  and 
cost  goals  for  this  firm’s  chosen  providers  were  driven  by  the  budget 
reduction. 


33  Some  firms  join  benchmarking  networks  with  similar  firms. 
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Open,  Ongoing  Communication  Helps 

Customer  Achieve  Desired  Sourcing  Results 


informal  Review 

Formal  Review 

•  Daily,  weekly 

•  Monthly,  quarterly, 
annually 

•  Project  manager  for 
both  customer  and 
provider 

•  Higher  leadership  for 
both  customer  and 
provider 

•  Discuss  ongoing 
work,  problems, 
upcoming  projects 

•  Monitor  specified 
performance  metrics,  set 
future  goals,  align 
workscope  with  budget 

•  Specified  in  contract 
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Once  the  customer  and  provider  have  chosen  performance  metrics  and 
associated  goals  to  manage  their  relationship,  they  need  a  process  to 
communicate  with  one  another  about  how  well  the  provider  is  meeting 
customer  needs.  The  communication  process  gives  the  customer 
opportunities  to  reinforce  good  performance,  draw  attention  to  problems,  and 
convey  any  changes  in  priorities  or  needs.  Moreover,  this  process  gives  the 
provider  opportunities  to  demonstrate  how  it  is  contributing  value  to  the 
customer. 

Most  of  the  customers  and  providers  of  facility  management  services 
that  we  interviewed  communicate  with  their  partners  through  both  an 
informal  and  a  formal  review  process.  Informal  reviews  occur  daily  or  weekly 
and  primarily  involve  the  project  managers  for  both  the  customer  and 
provider  organizations.  One  customer  literally  tore  down  the  walls  that 
separated  its  project  management  from  the  staff  of  its  provider  to  facilitate 
frequent  communication.  Project  managers  use  informal  reviews  to  discuss 
the  details  of  day-to-day  operations  such  as  ongoing  work,  the  status  of 
previous  initiatives,  new  problems  that  have  arisen,  “get-well”  programs  for 
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previous  problems,  and  upcoming  projects.  Customers  and  providers  are  able 
to  discuss  and  resolve  issues  before  they  become  costly  problems. 

Formal  reviews  occur  at  less  frequent  intervals,  e.g.,  monthly, 
quarterly,  or  annually,  and  may  include  higher  leadership  from  both  the 
customer  and  provider  organizations.^^  Two  providers  told  us  that  as  they 
demonstrate  continued  satisfactory  performance,  their  customers  lengthen 
the  time  between  their  formal  reviews. 

Many  customers  and  providers  stressed  the  importance  of  involving 
senior  leadership  in  managing  the  customer/provider  relationship.  On  the 
provider  side,  senior  management  needs  to  be  aware  of  how  well  their 
company  is  satisf5dng  customer  needs.  The  involvement  of  customer  senior 
management  provides  an  opportunity  to  tailor  facilities  services  to  the 
strategic  needs  of  the  firm. 

Formal  reviews  are  used  to  discuss  higher-level  issues  such  as  the 
provider’s  performance  against  a  specified  set  of  output-oriented  metrics,  to 
set  goals,  and,  if  necessary,  to  realign  the  scope  of  work  with  the  budget.  One 
large  customer  uses  monthly  reviews  to  monitor  performance  within  its 
regions  and  then  has  comprehensive  reviews  with  all  regions  quarterly. 
During  quarterly  reviews,  regional  providers  are  strongly  encouraged  to 
discuss  problems  in  individual  regions  and  share  best  practices. 

Although  the  details  of  the  information  covered  in  formal  reviews  may 
not  be  specified  in  the  contract  (e.g.,  specific  performance  metrics),  the  review 
process  is  often  spelled  out.  This  is  an  additional  way  that  customers  reduce 
the  risk  of  losing  control  over  the  provider  while  maintaining  the  flexibility  to 
shape  services  as  needs  change. 


^^Different  firms  have  formal  reviews  at  different  intervals.  One  provider  has  formal 
reviews  monthly  and  quarterly  with  its  customers  while  another  focuses  on  large 
comprehensive  annual  reviews.  High-level  corporate  leaders  are  often  involved  in  only  the 
quarterly  or  annual  reviews  rather  than  monthly  reviews. 
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Monitoring  Is  Costly  to  the  Customer 


m 


Customers  pay  for 

-all  information  used  to  monitor  providers 
-time  spent  examining  information 


I 


Customers  hurt  themselves  by  requesting 
information  that  is  not  used  to  make  decisions  or 
to  perform  other  activities 


I 
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Customers  may  have  a  strong  compulsion  to  track  many  different 
dimensions  of  operational  performance  and  cost,  feeling  that  this  is  necessary 
to  maintain  control  and  to  verify  that  their  providers  are  achieving  the 
agreed-upon  level  of  performance  within  the  specified  budget.  Although 
customers  are  responsible  for  monitoring  the  performance  and  cost  outputs, 
providers  note  that  customers  benefit  from  taking  into  consideration  the  fact 
that  they  pay  for  the  privilege  of  seeing  this  information,  regardless  of  which 
party  generates  it.  When  providers  are  required  to  generate  metrics  and 
reports,  they  pass  through  to  their  customers  the  costs  of  compiling  the 
information.  Similarly,  customers  pay  for  the  information  they  put  together 
internally  or  purchase  from  third-party  auditors.  Also,  there  is  an 
opportunity  cost  associated  with  the  time  customer  managers  spend 
examining  this  information. 

Innovative  customers  of  facility  management  services  carefully  select 
the  information  that  they  request  and  examine.  Customers  hurt  themselves 
by  asking  for  information  that  they  do  not  use  to  make  decisions  or  for  other 
activities.  During  discussions  with  its  customers  about  the  choice  of  metrics, 
one  provider  asks  how  each  metric  will  be  used.  It  discourages  customers 


-56- 


from  requesting  information  that  does  not  add  value,  net  of  generation  and 
opportunity  costs,  to  their  organizations. 

In  the  end,  as  noted  above,  the  actual  number  of  metrics  that  customers 
request  to  be  reported  depends  heavily  on  trust.  Trust  cannot  persist  in  a 
vacuum;  metrics  play  a  central  role  in  sustaining  trust.  But  because  metrics 
are  costly  to  use,  the  better  the  relationship  between  a  customer  and  provider 
is,  the  fewer  the  metrics  typically  used  to  sustain  a  productive  relationship. 
At  a  minimum,  a  customer  will  demand  information  on  the  satisfaction  of  the 
departments  and/or  business  units  within  its  organization  served  by  the 
provider.  Customer  satisfaction  is  typically  broken  down  by  service  provided; 
different  ratings  might  be  tracked  for  grounds  maintenance,  mailroom  and 
copying  services,  personnel  management  services,  and  so  on.  In  a  cost-based 
relationship,  customers  and  providers  always  track  disbursements  relative  to 
plan  and  cost  savings  relative  to  plan.  Quantitative  metrics  for  specific 
services  are  added  where  difficulties  persist  or  the  provider  is  explicitly 
trying  to  improve  performance  against  a  plan.  Relationships  between 
customers  and  providers  are  always  dynamic;  because  they  are  costly  to 
maintain,  the  actual  metrics  that  customers  request  vary  over  time  to  reflect 
experience  to  date  and  the  demands  of  joint  plans  to  change  processes  and 
performance. 
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We  conclude  our  discussion  of  the  uses  of  metrics  by  innovative 
commercial  firms  with  an  example  of  a  particular  provider’s  alignment  and 
management  process.  This  provider  uses  the  process  to  tailor  performance 
metrics  to  individual  customer’s  sites  and  circumstances  and  to  manage  the 
customer/provider  relationship. 

This  provider  prefers  that  the  alignment  process  begin  after  its  proposal 
has  been  selected  and  the  contract  signed.  First,  the  provider  and  customer 
sit  down  to  better  define  the  customer’s  expectations  through  a  vision  or 
mission  statement  for  the  project.  The  provider  then  asks  the  customer  to 
identify  the  dimensions  of  performance  that  are  keys  to  successfully 
accomplishing  the  mission  statement — Key  Result  Areas  (KRAs).  Examples 
include  ability  and  willingness  of  the  provider  to  respond  to  new 
requirements;  timeliness  of  completion  of  work;  cost  containment;  use  of 
technology;  training;  equipment  reliability;  and  use  of  small,  women-owned, 
or  minority-owned  businesses.  Customers  often  begin  by  la3nng  out  all  the 
dimensions  of  performance  that  they  care  about  and  then  refining  the  list  to 
four  or  five  major  areas  that  become  the  KRAs.  The  provider  stresses  the 
importance  of  going  through  this  exercise  at  all  levels  of  the  customer’s 
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organization  to  avoid  any  disconnects  between  the  expectations  of  top 
management  and  the  project  managers. 

Next,  for  each  of  the  KRAs,  the  customer  and  provider  discuss  the 
important  concerns  about  meeting  expectations  and  identify  four  or  five 
metrics  that  reflect  those  concerns.  For  example,  if  timeliness  of  work  is  a 
KRA,  then  one  type  of  metric  might  be  cycle  times  for  work  orders  and 
preventive  maintenance.  For  each  metric,  the  customer  and  provider  discuss 
how  the  metric  will  be  generated  and  how  it  will  be  reported  to  the  customer. 
Finally,  they  set  performance  goals  for  each  of  the  metrics  for  the  first  year  of 
the  contract.  In  the  case  of  cycle-time  metrics,  they  may  agree  upon  a  10 
percent  reduction  in  times  as  a  goal.  As  part  of  the  process  of  identifying 
goals,  the  customer  and  provider  discuss  potential  projects  that  will  help  the 
provider  achieve  the  goals.  For  example,  they  may  examine  the  possibility  of 
implementing  a  centralized  call  center  to  help  cut  cycle  times  for  work  orders. 

Throughout  this  process,  the  provider  stresses  to  the  customer  the  need 
to  ensure  that  the  quality  of  available  data  is  acceptable  and  that  each  metric 
yields  net  benefits. 
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Case  Study:  One  Provider’s  Alignment  and 

Management  Process  (Cont.)  _ 


.  Project  managers  for  provider  and  customer  meet 
informally,  daily,  weekly,  and  monthly 

.  Meet  annually  to  formally 

-  review  all  aspects  of  performance 

-  set  goals  for  coming  year 


rand  force  ■ 

Once  the  relationship  is  in  place,  this  provider  relies  upon  frequent 
interaction  between  its  staff  and  the  customer  to  make  sure  that  it  continues 
to  satisfy  the  customer’s  expectations  and  needs.  Project  managers  meet 
daily,  weekly,  and  monthly  to  informally  discuss  performance. 

In  addition  to  this  frequent  interaction  concerning  the  tactical  aspects  of 
the  relationship,  this  provider  encourages  each  of  its  clients  to  schedule  an 
annual  meeting  to  formally  discuss  the  strategic  dimensions  of  the 
relationship.  During  this  meeting,  the  provider  seeks  a  detailed  critique  of 
its  performance  that  it  and  the  customer  can  use  to  build  future  performance 
goals.  The  customer  and  provider  also  evaluate  the  process  and  metrics  that 
they  are  using  to  manage  the  relationship,  discuss  work  processes  in  place, 
and  examine  HR  practices  such  as  training.  When  possible,  the  provider 
encourages  the  customer  to  involve  corporate  leaders,  in  addition  to  project 
management,  in  this  review. 

Because  of  the  cost  of  conducting  an  in-depth  annual  review,  some 
customers  that  are  happy  with  this  provider’s  performance  request  to  forgo 
the  review.  However,  the  provider  tries  to  convince  them  to  go  through  the 
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process  because  it  is  an  integral  part  of  the  provider’s  continuous 
improvement  process. 
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6.  IMPLICATIONS  FOR  THE  AIR  FORCE 


.  The  many  roles  of  performance  metrics  in  the 
sourcing  process 

.  Choosing  the  appropriate  set  of  metrics 
.  Coiiecting  data  and  generating  metrics 
.  Managing  performance  and  cost  against  goais 
.  Implications  for  the  Air  Force 
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The  process  of  choosing  and  managing  sources  for  support  services  is 
not  new  to  the  Air  Force.  Many  of  the  ideas  presented  in  the  previous 
chapters  differ  dramatically  from  the  way  the  Air  Force  has  approached  its 
sourcing  process  in  the  past.  However,  they  are  broadly  compatible  with  new 
and  ongoing  Air  Force  efforts  to  extend  the  benefits  of  acquisition  reform  to 
services  acquisition  and,  in  particular,  to  introduce  performance-based 
services  acquisition  (PBSA).  Without  an  effective  regime  of  metrics,  PBSA  is 
a  meaningless  concept. 

The  traditional  Air  Force  approach,  which  was  based  on  the  Office  of 
Federal  Procurement  Policy,  began  with  the  identification  of  a  required 
activity  or  service.^  Once  the  customer  identified  the  requirement,  the 
contracting  officer  ran  the  source  selection  process.  The  statement  of  work 
reflected  a  static  view  of  this  requirement,  specifying  the  desired  performance 
attributes  (based  on  current  practice)  and  their  relative  importance.  Metrics 

35This  approach  was  similar  to  that  of  other  military  services  and  federal  agencies. 
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often  tracked  performance  of  processes  or  application  of  tools  rather  than  the 
outputs  of  those  processes  or  tools.  They  specified  the  performance  of  these 
processes  in  such  detail  that  process  improvement  was  inhibited  or  even 
prohibited.  Goals  for  performance  metrics  typically  represented  the 
minimum  acceptable  level  of  performance  to  avoid  unnecessary  exclusion  of 
any  firms  from  the  set  of  potential  providers.  Past  performance  and 
experience  were  not  taken  into  account  in  the  source  selection  process.  If  an 
external  provider  was  chosen,  that  provider’s  main  interaction  with  the  Air 
Force  took  place  through  Quality  Assurance  Evaluators  (QAEs)  who  are 
trained  to  scrutinize  the  provider’s  actions  to  make  sure  that  it  is  meeting  all 
the  contract  specifications.  As  a  result  of  the  high  cost  of  bidding,  the  low 
probability  of  winning  relative  to  strong  commercial  demand,  and  the  arm’s- 
length  nature  of  the  resulting  customer/provider  relationship,  many  leading 
providers  chose  not  to  compete.  This  traditional  approach  has  led  to  failures 
to  perform  and  cost  escalation  in  many  instances  because  poor  providers  have 
been  awarded  contracts  based  on  proposals  that  they  cannot  execute. 

Federal  procurement  policy  has  experienced  dramatic  changes  in  recent 
years,  primarily  as  a  result  of  the  Federal  Acquisition  Streamlining  Act 
(FASA)  of  1994.  Perhaps  the  most  dramatic  change  is  the  new  emphasis  on 
performance-based  contracting.  The  Federal  Acquisition  Regulation,  Part  37, 
on  Service  Contracting  defines  performance-based  contracting  as  “structuring 
all  aspects  of  an  acquisition  around  the  purpose  of  the  work  to  be  performed 
as  opposed  to  either  the  manner  by  which  the  work  is  to  be  performed  or 
broad  and  imprecise  statements  of  work”  (section  37.101). 

These  changes  have  empowered  the  Air  Force  to  introduce  PBSA.  Past 
performance  is  being  added  to  the  source  selection  criteria,^®  formal  metrics 
in  contracts  are  becoming  more  output  oriented,  and  threshold  levels  of 
performance  and  cost  are  being  tied  to  positive  and  negative  incentives.  The 

36  Very  little  past  performance  data  on  government  contractors  exist  at  this  time.  The 
data  that  have  been  collected  are  not  readily  available  to  customers  across  Air  Force 
organizations,  and  it  will  be  some  time  before  information  on  current  contracts  is  widely 
available.  However,  information  on  past  performance  should  be  aveulable  from  commercial 
customers  of  providers.  Relatively  more  information  is  available  for  extreme  cases  of  poor 
performance  on  government  contracts.  Providers  must  report  whether  they  have  defaulted 
on  a  government  contract  within  the  last  three  years  when  bidding  on  a  new  government 
contract. 
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innovative  commercial  practices  discussed  above  provide  illustrations  that 
the  Air  Force  can  use  to  (1)  move  toward  PBSA  and  (2)  over  the  longer  term 
move  beyond  it  toward  commercial-like  service  acquisition  practices  that  will 
allow  even  higher  performance  levels  and  encourage  even  more  of  the  best 
commercial  firms  to  provide  services  to  the  Air  Force. 

The  Air  Force  cannot  do  this  alone.  Broader  public  policy  will  have  to 
advance  to  accommodate  all  the  practices  discussed  here.  In  this  final 
chapter,  we  discuss  how  the  Air  Force  can  apply  many  of  the  practices  of 
innovative  commercial  customers  and  providers  to  achieve  better  results  fi'om 
its  sourcing  activities  today  and  promote  more  effective  public  policies  on 
services  acquisition  in  the  future.  We  provide  information  about  how  the  Air 
Force  might  implement  these  practices  within  its  current  environment; 
however,  more  research  is  needed  to  fully  address  any  potential  challenges. 
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Air  Force  Can  Use  Metrics  to  Improve 
Sourcing  Decisions 


When  deciding  whether  to  conduct  a  sourcing  study, 

•  Metrics  can  help  Air  Force  identify  best  available 
performance/cost 

-third-party  audits  can  help  baseline  internal 
performance/cost 

-commercial  benchmarking  studies  identify 
world-class  performance/cost 

When  selecting  a  provider, 

•  Metrics  help  clarify  dimensions  of  performance 
that  best  promote  strategic  goals 
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Performance  metrics  can  help  the  Air  Force  make  better  sourcing 
decisions  by  helping  it  become  more  aware  of  the  capabilities  of  external 
providers  and  how  they  compare  with  internal  organic  provider 
organizations.  This  increased  awareness  can  lead  to  a  better  allocation  of 
scarce  sourcing  study  resources  and  improved  source  selection  decisions 
between  internal  and  external  provision  of  services  and  among  external 
providers. 

The  Air  Force  has  completed  an  assessment  of  its  commercial  activities 
that  are  currently  performed  by  military  or  government-employed  civilians  to 
determine  whether  they  are  critical  to  the  warfighting  mission  or  candidates 
for  competitive  sourcing.^^  For  those  installation  services  that  are 
candidates,  the  Air  Force  has  two  vehicles  through  which  to  conduct  sourcing 
studies:  competition  under  the  rules  of  Office  of  Management  and  Budget 
(0MB)  Circular  A-76  and  direct  conversion  from  military  to  contractor 

^  The  Air  Force  recently  identified  54,000  potentied  positions  to  compete  between 
FY93  and  FY03.  Over  22,000  of  those  positions  are  scheduled  for  competition  between  FYOO 
and  FY03  (Bundy,  1999). 
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provision.^  All  candidates  cannot  be  competed  at  once  because  of  resource 
constraints  and  personnel  disruptions,  so  the  Major  Commands  are  selecting 
activities  to  be  competed  each  year.  This  selection  process  appears  to  suffer 
from  a  lack  of  information  about  the  best  candidate  activities.  Functionals, 
who  have  the  most  information  about  the  savings  potentials  associated  with 
competing  different  activities,  have  little  incentive  to  participate  effectively  in 
this  process. A  lack  of  information,  combined  with  other  issues  such  as 
socioeconomic  concerns,  can  lead  the  Air  Force  to  allocate  study  resources 
suboptimally. 

Instead,  the  Air  Force  can  use  performance  metrics  to  identify  the  best 
performance  and  cost  that  are  available  in  the  external  market  and  to 
compare  performance  and  cost  in  the  external  market  with  those  of  current 
organic  providers.  This  information  could  help  the  Air  Force  direct  scarce 
study  resources  to  those  services  and  activities  that,  ex  ante,  appear  to  offer 
the  largest  potential  benefits.^  However,  such  a  public/private  comparison  is 
not  easy  because  of  a  lack  of  information  about  organic  processes  and 
performance  as  well  as  difficulties  in  identif5dng  “apples-to-apples” 
comparisons  with  external  capabilities. To  overcome  these  difficulties,  the 
Air  Force  can  use  third-party  operational  and  financial  audits  to  baseline  the 
performance  and  costs  associated  with  its  organic  providers.^  In  addition,  it 
can  use  commercial  benchmarking  studies  to  identify  the  performance  and 

^  There  are  separate  rules  that  govern  sourcing  competitions  for  depot  maintenance 
activities. 

39See  Baldwin,  Camm,  Keating,  and  Pint  (1998)  for  a  discussion  of  Air  Force 
incentives  to  imdertake  and  complete  sourcing  studies. 

■^^Large  differences  between  external  and  organic  provision  do  not  imply  that  a 
sourcing  study  will  result  in  external  provision.  In  many  circumstances,  the  orgamic  Most 
Efficient  Organization  (MEO),  which  is  the  civilian  organization’s  bid,  reflects  significant 
cost  improvements  over  the  current  organization.  Early  research  by  the  Center  for  Naval 
Analyses  (CNA)  indicates  that  the  Navy  achieved  almost  20  percent  savings  through  its 
studies  won  by  the  MEO  (Marcus,  1993).  Recent  RAND  research  foimd  that  expected 
personnel  cost  savings  for  three  competitions  won  by  the  MEO  range  from  34  to  59  percent 
(Gates  and  Robbert,  in  draft). 

^^As  discussed  above,  the  Air  Force  is  not  the  only  organization  facing  such 
challenges.  Many  commercial  firms  encountered  similar  obstacles  and  successfully  overcame 
them. 

The  City  of  Indianapolis  also  lacked  information  about  the  cost  of  performing 
activities  and  services  when  it  began  its  outsourcing  activities.  It  hired  KPMG  Peat  Marwick 
to  develop  cost  models  and  to  train  its  administrators  to  use  them  to  implement  activity- 
based  costing.  See  Goldsmith  (1997). 


cost  associated  with  world-class  commercial  providers.  For  BOS  (and  audit) 
services,  it  is  important  to  look  beyond  traditional  defense  contractors  to  the 
set  of  providers  that  are  known  as  leaders  in  the  commercial  market  for 
facility  management  services.  Access  to  benchmarking  studies  can  be 
obtained  through  Air  Force  participation  in  such  studies  and  through  trade 
associations  such  as  the  International  Facility  Management  Association 
(IFMA),  Building  Owners  and  Managers  Association  (BOMA),  and  Tradeline. 

If  the  Air  Force  decides  to  study  a  service  or  activity,  metrics  and  their 
associated  weights  can  aid  in  the  selection  of  the  provider  organization  that 
best  meets  the  unique  needs  of  the  customer  organization  (e.g.,  an 
installation,  a  Major  Command,  the  Air  Staff).  In  fact,  in  many  situations, 
new  federal  acquisition  policy  requires  federal  agencies  to  use  source  selection 
criteria  that  reflect  key  dimensions  of  performance  in  addition  to  the  cost 
associated  with  the  scope  of  work.  (See,  for  example,  FAR  12.206  and 
12.602.)  The  customer  organization  can  issue  Requests  for  Information 
(RFIs)  to  all  potential  bidders  to  better  understand  capabilities  of  potential 
providers.  This  additional  information  can  be  used  to  refine  the  selection 
criteria,  or  metrics.  The  weights  associated  with  these  metrics  should  be 
chosen  by  the  Air  Force  customer  that  will  actually  consume  any  service 
provided,  with  advice  and  recommendations  from  contracting  and  relevant 
functional  personnel.  The  relevant  Air  Force  customer  is  typically  an 
operational  Air  Force  organization,  e.g.,  a  wing  or  squadron,  or  an 
organization  that  represents  the  military  personnel  and  their  dependents 
who  consume  base-related  services.  The  use  of  metrics  and  associated 
weights  in  the  source  selection  process  helps  focus  selection  decisions  on 
those  dimensions  of  performance  that  directly  contribute  to  the  strategic 
goals  of  the  Air  Force  customer  organization. 

As  discussed  above,  the  use  of  performance  metrics  in  the  source 
selection  decision  can  also  help  convince  senior  Air  Force  leaders,  such  as 
wing  commanders,  and  Air  Force  personnel,  such  as  the  functional 
leadership,  that  the  chosen  provider  will  best  meet  the  needs  of  the  Air  Force. 
The  objective  nature  of  performance  metrics  can  help  create  a  feeling  of 
legitimacy  in  a  process  that  consumes  scarce  resources  and  leadership  focus 
and  is  often  emotional. 
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Keep  detailed  management  metrics  and  goals 
outside  contract  so  they  can  be  adjusted  easily 
over  time 


-change  in  wing  commander 
-  budget  cuts 


Communicate  frequently  with  provider 

-  involve  customer  and  provider  managers  at  all 
levels  of  organizations 
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If  the  Air  Force  chooses  to  outsource  a  given  service  or  activity  and 
selects  a  provider  with  a  successful  track  record,  it  can  use  performance 
metrics  to  better  manage  the  customer/provider  relationship  and  achieve 
more  desirable  performance  outputs  throughout  the  length  of  the  contract. 

Metrics  used  to  manage  the  relationship  between  the  Air  Force  and  one 
of  its  providers  should  focus  on  the  outputs  of  services  and  activities,  rather 
than  the  processes  used  to  achieve  them.^s  The  closer  Air  Force  oversight 
gets  to  the  way  the  provider  operates,  the  greater  the  chance  that  the  Air 
Force  will  deprive  itself  of  the  benefits  of  innovation  in  processes  and 
technologies  that  led  to  the  selection  of  that  provider.  At  the  heart  of  the  Air 
Force’s  ongoing  effort  to  implement  PBSA  is  the  challenge  of  moving  from 
process-oriented  to  output-oriented  metrics  to  manage  providers.^  For 

^  Of  course,  if  the  performance  of  the  provider  falls  below  customer  expectations,  the 
Air  Force  may  want  to  work  with  the  provider  to  resolve  difficulties.  Such  collaboration  may 
require  the  use  of  additional  process-oriented  metrics.  Process-oriented  metrics  may  also  be 
desired  for  those  activities  and  services  with  outputs  that  are  difficult  to  specify. 

^  Many  government  agencies  are  facing  similar  difficulties.  The  GAO  (1997)  foimd 
that  the  most  challenging  dimension  of  implementing  the  Government  Performance  and 
Results  Act  of  1993  is  moving  from  measuring  activities  to  measuring  the  results  of  those 
activities.  See  also  GAO  (1999). 
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example,  a  typical  Air  Force  contract  for  cleaning  restrooms  now  specifies  the 
number  of  times  the  restrooms  must  be  cleaned  each  week,  rather  than  the 
process  used  to  clean  them.  Although  this  is  a  step  in  the  right  direction, 
customer  satisfaction  with  cleanliness  or  a  cleanliness  rating  by  a  third-party 
auditor  is  closer  to  a  true  output-oriented  metric.  Although  new  federal 
acquisition  policy  supports  this  shift  toward  output-oriented  metrics, 
traditional  Air  Force  culture  and  organizational  structure  have  supported  an 
input  rather  than  an  output  orientation,  which  will  be  difficult  for  the 
sourcing  community  to  overcome.^ 

The  Air  Force  collects  many  metrics  to  help  it  monitor  the  activities  of 
its  contractors.  Although  some  of  these  metrics  contribute  to  decisionmaking 
or  other  activities,  many  are  never  acted  on.*^  As  the  Air  Force  achieves 
higher  levels  of  mutual  trust  with  its  providers,  it  can  benefit  from  the  use  of 
a  smaller  set  of  meaningful  metrics  that  focuses  attention  on  those 
dimensions  of  performance  that  most  directly  contribute  to  the  Air  Force’s 
corporate  strategic  goals.  In  addition,  by  evaluating  the  net  value  that 
metrics  add — ^taking  into  account  the  costs  of  generating  and  reporting  them 
as  well  as  the  staff  time  required  to  monitor  them — the  Air  Force  can  reduce 
its  monitoring  costs. 

The  Air  Force  can  benefit  from  separating  the  choice  of  performance 
metrics  and  goals  used  to  manage  a  given  customer/provider  relationship 
from  the  associated  contract.  As  a  result,  metrics  can  be  changed  more  easily 
as  customers  and  providers  learn  about  meeting  customer  needs.  Air  Force 
customers  traditionally  expect  contracting  to  reduce  their  flexibility,  because 
a  signed  contract  generates  a  “must-pay”  bill  over  its  lifetime  and  its  quality 
assurance  surveillance  plan  states  what  is  expected  over  its  lifetime. 

Changes  can  be  made  only  through  a  formal  contract  modification.  But  in  the 
current  environment,  the  Air  Force  perceives  a  need  for  more  flexibility,  not 
less.  And  with  the  recent  trend  toward  longer-term  contracts,  it  is 

Training  in  best  commercial  practices  for  performance  measurement  and  exposure 
to  exemplar  commercial  firm  ex3imples  through  forums  such  as  Tradeline  conferences  can 
help  address  this  need. 

Some  metrics  may  be  collected  primarily  because  the  data  are  readily  available. 
Baker  (1992)  suggests  that  when  organizational  objectives  are  not  easily  quantifiable, 
organizations  may  employ  targets  that  are  easily  measured  but  that  may  not  be  highly 
correlated  with  the  organization’s  true  objectives. 


-69- 


increasingly  likely  that  wing  commanders  will  rotate  at  least  once  during  a 
contract  period,  bringing  about  changes  in  customer  priorities  for  BOS 
services.  Leaving  metrics  and  goals  out  of  the  contract  allows  each  wing 
commander  more  flexibility  to  shape  services  at  his  or  her  installation. 
Similarly,  changes  in  budgets  (which  are  becoming  more  and  more  common) 
can  be  better  managed  when  metrics  and  goals  are  not  tied  to  formal 
contracts.  Thus,  by  leaving  metrics  and  goals  outside  of  the  formal  contract, 
the  Air  Force  customer  and  the  provider  can  more  easily  satisfy  Air  Force 
priorities  at  any  given  time.^'^  The  most  innovative  commercial  providers  of 
support  services  are  comfortable  with  such  an  approach.  The  Air  Force’s  new 
policy  governing  contracting  for  services,  AFI 63-124,  allows  Air  Force 
customer  organizations  to  choose  metrics  and  goals  after  contract  award,  but 
before  the  contract  start  date. 

Because  the  Air  Force  is  limited  in  its  ability  to  provide  informal 
incentives  to  its  providers  (e.g.,  expanding  contract  scope  without  a  rebid),  it 
must  rely  on  formal  contractual  incentives  (e.g.,  award  fees)  to  motivate  its 
providers  to  excel  over  time.*^  However,  separating  metrics  from  the  contract 
breaks  the  link  between  metrics  and  formal  incentives,  which  is  at  the  heart 
of  performance-based  acquisition.  The  Air  Force  can  preserve  flexibility  and 
still  give  providers  incentives  by  including  in  contracts  only  those  metrics 
that  are  tied  to  formal  award  fees  and  by  keeping  all  other  management 
metrics  outside  of  contracts.^^  Overarching  metrics  such  as  customer 
satisfaction  ratings  are  ideal  for  formal  incentives  because  their  importance 
is  unlikely  to  change  during  the  contract  period  and  they  are  less  likely  to 
lead  to  the  distortions  in  provider  performance  discussed  above. 

Finally,  the  Air  Force  can  improve  the  results  of  its  sourcing  activities 
by  communicating  openly  and  frequently  with  its  providers.  Currently,  the 

47  Courty  and  Marschke  (1997)  suggest  that  it  is  more  difficult  for  government 
agencies  to  continually  reevaluate  and  realign  metrics  with  org2inizational  goals  because 
often  there  is  no  specific  person(s)  who  directly  benefits  from  this  process. 

48  As  discussed  above,  another  informal  incentive  used  in  the  private  sector  is  contract 
extension  without  a  rebid  and  without  specification  of  decision  criteria  in  the  original 
contract.  The  Air  Force  has  transformed  this  idea  into  a  formed  incentive  through  the  use  of 
“award  terms”  that  extend  existing  contracts  based  on  provider  performance  relative  to 
standards  specified  in  those  contracts. 

49  Dan  Fulmer  of  HQ  AFMC/AQ  indicates  that  if  additional  metrics  are  specified  in 
the  requirements  section  of  the  bid  solicitation,  they  must  be  included  in  the  contract  as  well. 
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Air  Force’s  primary  communication  with  its  providers  occurs  through  its 
QAEs.  This  method  of  review  and  feedback  reinforces  an  adversarial,  arm’s- 
length  working  environment  between  the  Air  Force  and  its  providers,  rather 
than  building  a  mutually  beneficial  relationship  based  on  open 
communication  and  trust.  In  contrast,  daily  and  weekly  communication 
between  the  project  managers  or  leaders  of  the  customer  organization  (rather 
than  the  QAEs)  and  the  provider  would  help  build  understanding  and  trust 
that  can  lead  to  more  responsive  service  for  the  Air  Force.  To  build  trust,  one 
goal  of  such  communication  should  be  to  identify  and  solve  problems  before 
they  escalate,  rather  than  to  look  for  reasons  to  penalize  the  provider. 
(Unfortunately,  some  interpretations  of  performance-based  acquisition  still 
focus  on  the  latter.)  Another  goal  should  be  to  respond  quickly  to  unexpected 
workscope  or  budget  events.  Providers  are  often  willing  to  make  suggestions 
about  the  best  way  to  address  unexpected  events,  but  customer  input  is 
necessary  to  ensure  that  any  changes  in  services  continue  to  support 
strategic  goals.  High-level  management  on  both  sides  should  also  participate 
in  the  communication  process,  on  a  quarterly  to  annual  basis  as  needed. 
Depending  on  the  organizational  levels  covered  in  the  contract, 
communication  between  wing  commanders.  Major  Command  personnel,  and 
even  Air  Staff  leadership  and  the  provider’s  corporate  leaders  can  improve 
the  contributions  of  the  provider  to  the  strategic  goals  of  the  Air  Force. 

If  the  Air  Force  (1)  uses  metrics  that  are  aligned  with  its  strategic 
goals,  (2)  sets  aggressive  goals  for  the  metrics,  and  (3)  adopts  an  approach  to 
performance  management  that  promotes  day-to-day  communication  and 
problem  solving  and  longer-term  mutual  efforts  to  improve  processes,  best 
commercial  practice  suggests  that  the  Air  Force  will  enhance  and  sustain 
mutual  trust  with  its  service  providers  and  thereby  dramatically  reduce  the 
manpower  and  other  costs  associated  with  quality  assurance.  Good  metrics 
are  powerful  tools  that  allow  a  small  group  of  project  managers  to  track  the 
important  dimensions  of  a  provider’s  performance.  For  example,  a  large 
customer  firm  that  has  outsourced  35  million  square  feet  of  facilities  has  five 
or  six  people  dedicated  only  part-time  to  managing  its  providers.  As  Air 
Force  leaders  feel  more  comfortable  with  their  ability  to  maintain  control  of 
external  service  providers,  the  Air  Force  may  choose  to  turn  over  more  of  the 
management  of  services  to  them.  Our  interviews  with  commercial  firms 
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indicate  that  the  resulting  partnership  environment  can  lead  to  much  greater 
improvements  in  performance  and  cost  over  time.^o 

^  See  also  Goldsmith  (1997). 
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Many  of  the  Air  Force’s  activities  will  continue  to  be  provided  by  organic 
workforces  in  the  foreseeable  future.  Activities  considered  too  important  to 
the  warfighting  mission  are  removed  from  the  list  of  possible  candidates  for 
sourcing  studies.  It  will  be  some  time  before  some  other  candidate  activities 
will  be  evaluated  for  possible  outsourcing  because  of  the  time,  resources,  and 
disruptions  associated  with  studies. 

Traditionally,  the  Air  Force  has  placed  greater  emphasis  on  measuring 
the  performance  of  its  external  providers  than  on  measuring  its  organic 
providers.  Unfortunately,  this  lack  of  information  about  processes  and 
outputs  of  organic  activities  limits  the  ability  of  Air  Force  personnel  to 
improve  their  performance  and  cost.  A  key  insight  of  this  research  is  that 
many  of  the  ways  innovative  firms  use  performance  metrics  to  manage 
customer/provider  relationships  can  be  applied  to  internal  provider 
organizations  as  well  as  external  ones.  Just  as  the  Air  Force  can  use 
performance  metrics  to  improve  the  outputs  associated  with  its  outsourcing 
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activities,  it  can  use  metrics  to  improve  the  performance  of  its  internal 
organic  provider  organizations. 

In  particular,  the  Air  Force  can  use  output-oriented  metrics  to 
benchmark  performance  and  costs  across  its  own  sites  and  to  compare  them 
against  the  performance  and  costs  achieved  by  leading  commercial  firms. 
Comparisons  of  customer  satisfaction  ratings,  financial  metrics,  and 
operational  metrics  can  be  used  to  continually  reassess  the  benefits  of 
keeping  a  service  or  activity  in-house. 

Organic  provider  organizations  can  use  process-oriented  metrics  to 
improve  their  performance  and  cost  over  time.  As  part  of  a  continuous 
improvement  program,  process-oriented  metrics  can  be  used  to  understand 
variances  identified  in  benchmarking  exercises  with  leading  commercial 
firms  and  with  other  military  and  government  organizations.  When 
variances  result  from  process  innovations,  the  innovations  can  be  shared 
across  Air  Force  sites  through  management  meetings,  web  sites,  and  other 
communication  tools. 

It  is  important  to  note  that  simply  having  output  and  process  metrics  is 
not  sufficient  to  bring  about  improved  performance  and  cost  of  an  organic 
provider.  Air  Force  customer  and  provider  organizations  must  jointly  set 
aggressive  performance  goals  for  those  metrics,  measure  performance  against 
those  goals  regularly,  and  then  hold  organic  provider  personnel  responsible 
for  meeting  the  goals.  Although  civil  service  employment  rules  and  reduced 
opportunities  for  advancement  in  the  current  environment  may  somewhat 
limit  the  Air  Force’s  ability  to  use  rewards/penalties  for  good/poor 
performance  by  civilian  workforces,  the  Air  Force  can  create  nonmonetary 
incentives,  such  as  promotion  opportunities,  for  its  military  workforces. 

Wilson  (1989),  Courty  and  Marschke  (1997),  Dixit  (1997),  and  Heckman  et  al. 

(1997)  discuss  the  difficulties  associated  with  providing  performance  incentives  to  employees 
of  government  agencies.  A  primary  driver  of  these  difficulties  is  that  there  are  no  private- 
sector  substitutes  for  many  of  the  activities  performed  by  these  organizations.  Thus,  it  is 
more  difficult  to  specify  and  quantify  objectives  so  that  performance  can  be  measiured. 
However,  this  is  clearly  not  the  case  for  most  BOS  services. 

52  Benchmarking  across  Air  Force  sites  should  be  done  constructively.  Installation 
commanders  may  not  wish  to  share  information  with  other  sites  if  they  perceive  that  the  risk 
of  being  penalized  for  relatively  poor  performance  and  costs  is  greater  than  the  potential 
benefits  from  learning  about  performance  in  other  organizations. 
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Finally,  to  benefit  from  the  use  of  such  metrics,  the  Air  Force  may  need 
to  invest  in  a  commercial  MIS  capability  or  outsource  its  facility  call  center 
operations  to  a  provider  with  a  robust  MIS  capability.  Currently  available 
Air  Force  data  may  not  be  sufficient  to  determine  outputs  or  to  understand 
processes  associated  with  the  provision  of  services. 
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Appendix  A 

INTERVIEW  QUESTIONNAIRE  ON  PERFORMANCE  METRICS 


In  advance  of  our  interviews  with  commercial  customers  and  providers 
of  facility  management  services,  we  provided  each  firm  with  a  structured 
questionnaire  detailing  the  dimensions  of  outsourcing  that  we  were  most 
interested  in  learning  about.  Some  firms  gave  us  written  responses  to  these 
questions;  others  used  them  as  a  guide  for  preparing  personnel  for  our  visit. 
In  each  case,  the  questions  served  as  an  effective  guide  for  our  discussions. 
The  customer  and  provider  questions  on  the  use  of  performance  metrics  are 
given  below. 

Customer  Questions 

A.  For  each  activity  you  outsource,  what  metrics  do  you  and  your  providers 
use  to  measure  performance,  and  how  frequently  are  they  measured 
(e.g.,  responsiveness,  reliability,  quality, . .  .)? 

1.  What  goals  or  standards  are  used  for  each  metric? 

B.  Which  metrics,  goals,  and  standards  are  explicitly  written  into  your 
contracts  and  which  are  negotiated  after  the  contract  is  complete? 

How  are  these  goals  typically  set? 

1.  What  is  your  role? 

2.  What  is  the  role  of  your  providers? 

C.  On  average,  how  many  metrics  do  you  use  to  evaluate  a  provider’s 
performance  for  a  given  activity? 

D.  In  general,  when  developing  metrics,  do  you  focus  on  the  outputs  of 
services  provided,  processes  involved  in  providing  services,  or  both? 

E.  Do  you  or  your  provider(s)  use  formal  (e.g.,  ISO  9000)  or  informal  audits? 
If  so,  do  you  use  them  for  measurement?  For  improvement?  For 
communication? 
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Provider  Questions 


A.  For  each  service  you  provide,  what  metrics  do  you  and  your  customers 
use  to  measure  performance,  and  how  frequently  are  they  measured 
(e.g.,  responsiveness,  reliability,  quality, . .  .)? 

1.  Can  you  rank  these  metrics  as  to  whether  they  are  always, 
often,  sometimes,  or  rarely  used? 

2.  What  goals  or  standards  are  typically  used  for  each  metric? 

3.  What  are  some  of  the  reasons  metrics  and  goals  vary  among 
customers  (e.g.,  industry,  size,  experience)? 

4.  In  general,  when  developing  metrics,  do  you  focus  on  the 
outputs  of  services  provided,  processes  involved  in  providing 
services,  or  both? 

B.  Which  metrics,  goals,  and  standards  are  explicitly  written  into  contracts 
and  which  are  negotiated  after  the  contract  is  complete? 

1.  How  are  goals  t5rpically  set? 

a.  What  is  your  role? 

b.  What  is  the  customer’s  role? 

C.  On  average,  how  many  metrics  are  you  evaluated  against  for  a  given 
activity? 

D.  Do  you  or  your  customers  use  formal  (e.g.,  ISO  9000)  or  informal  audits? 
If  so,  do  you  use  them  for  measurement?  For  improvement?  For 
communication? 
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Appendix  B 
METRICS  EXAMPLES 


In  this  appendix,  we  provide  examples  of  metrics  that  customers  use  to 
select  their  providers  and  metrics  that  customers  and  providers  use  to 
manage  their  relationship  over  time.  These  lists  are  not  inclusive.  Rather, 
they  provide  additional  concrete  examples  to  supplement  the  ideas  in  the 
main  text.  The  superscripts  after  each  metric  refer  to  the  sources  listed  at 
the  end  of  this  appendix. 

SOURCE  SELECTION  METRICS 

•  Total  facilities  cost  and  cost  of  specific  services  per  square  foot^ 

•  Percentage  of  a  provider’s  business  a  prospective  customer  would 
represent^ 

•  Depth  of  training  for  provider  employees  (how  well  trained  is  the 
second  string?)^ 

•  Financial  stability  of  provider^ 

•  Training  hours  (total  and  hours  per  employee)^ 

•  Percentage  of  business  done  with  small  or  minority-  or  women- 
owned  businesses^ 

•  Quality  of  environmental  and  safety  programs^?^ 

•  Employee  turnover  rate^ 

•  Reputation  relevant  to  ethical  business  practices^ 


METRICS  USED  TO  MANAGE  THE  CUSTOMER/PROVIDER 
RELATIONSHIP 

Call  Center  Metrics 

•  Number  of  customer  service  representatives  on  duty^ 

•  Total  number  of  calls^ 

•  Number  of  calls  per  customer  service  representative^ 
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•  Calls  answered  per  day^ 

•  Calls  abandoned  per  day<^ 

•  Average  queue  time  for  calls^ 

•  Average  number  of  calls  in  the  queued 

•  Total  queue  hours  ^ 

•  Average  duration  of  calls^ 

•  Average  hold  times^’l 

Customer  Satisfaction  Metrics 

•  Overall  customer  satisfaction®*^ 

•  Satisfaction  with  temperature,  janitorial  service,  and  landscaping® 

•  Satisfaction  with  quality  of  facility  support  staff,  ability  to  solve 
facility  problems,  availability,  timeliness  of  response,  follow-up  and 
closure  of  service  request,  courteousness,  and  understanding 
customer  needs  and  requirements®*^ 

•  Employee  satisfaction  with  building  services®*^ 

•  Owner  satisfaction  with  building  services^ 

•  Completion  of  scheduled  visits  (actual  number  of  tenant  visits 
divided  by  the  scheduled  number)^ 

•  Satisfaction  with  quality  of  space,  appearance,  configuration®*^*^ 

•  Satisfaction  with  efficiency  of  space® 

•  Most/least  desirable  aspects  of  property® 

•  Distribution  of  responses  indicating  that  service  exceeded,  met, 
nearly  met,  or  missed  customer  standards  (by  service)*^ 

•  Rating  of  responsiveness  and  demonstrated  flexibility  of  provider®-*^ 

Operations 

•  Total  cost  of  ownership  (includes  attention  required  by  customer 
managers,  continuous  improvement  programs)® 

•  Occupancy  cost  (total  and  by  line  item)® 

•  Cost  of  activities  relative  to  leading  firms  in  benchmarking  studies® 
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•  Total  facilities  cost  and  cost  of  specific  services  per  square  foot  (by 
space  classification,  market  location,  line  of  business)^^ 

•  Operating  expenses  per  real  estate  square  foots 

•  Occupancy  cost  per  square  foot,  by  location®’^ 

•  Building  maintenance  cost  per  gross  square  foot  vs.  customer 
satisfaction^ 

•  Total  annual  administrative  costs  per  square  foot^ 

•  Total  annual  cost  of  insurance  and  taxes  per  square  foot^ 

•  Variance  between  actual  expenses  and  budget  for  main  cost 
categories,  year  to  date  (in  dollars  and  as  a  percentage  of  budget)! 

•  Energy  cost  per  square  footS’! 

•  Total  utility  cost  per  square  footb! 

•  Occupancy  cost  per  output!^ 

•  Occupancy  ratio  (cost  as  a  percentage  of  revenue)!’!^ 

•  Cost  per  employee  (by  market  location,  line  of  business)!»!^ 

•  Capital  cost  per  employee^ 

•  Groundskeeping  cost  per  acre^ 

•  Total  custodial  cost  per  square  foot  cleaned  vs.  customer 
satisfaction^ 

•  Number  of  unscheduled  custodial  requests  per  month  by  service 
type! 

•  Number  of  unscheduled  custodial  requests  per  month  per  100,000 
square  feet! 

•  Cleaning  efficiency  (square  feet  cleaned  per  manhour)?^! 
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•  Cost  of  employee  moves 

•  Number  of  moves  per  yearJ 

•  “Churn”  rate  (average  moves  per  employee  per  year  or  time 
between  moves)^ 

•  Percentage  uptime  for  elevators^ 

•  Number  of  unscheduled  elevator  service  requests  per  month  per 
100,000  square  feet^ 

•  Number  of  unscheduled  elevator  service  requests  per  month  by  t3q)e 
of  problem^ 

•  Percentage  of  time  computer  systems  are  operational^ 

•  Total  work  orders,  by  task  and  by  priorit}^’^ 

•  Number  of  service  requests  per  month  by  siteS’^ 

•  Percentage  of  work  orders  initiated  by  customer® 

•  Work  order  and  inquiry  response  time®^>l 

•  Work  order  completion  timef’^’l 

•  Number  of  open  work  orders^’^ 

•  Frequency  of  work  orders  for  specific  pieces  of  equipment^ 

•  Labor  hours^^ 

•  Emergency  inquiry  response  time^ 

•  Security  incidents  per  month  per  100,000  square  feet^ 

•  Number  of  security  incidents  by  type^ 

•  Percentage  of  time  security  systems  are  operational^ 

•  Preventive  maintenance  tasks  completed  and  not  completed? 

•  Percentage  of  preventive  maintenance  completed  on  time?>^ 

•  Percentage  of  life  safety  preventive  maintenance  tasks  completed 
on  time^ 
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•  Number  of  service  requests  in  and  out  of  scopeS 

•  Number  of  unscheduled  maintenance  calls  per  100,000  square  feet 
(total  and  by  type  of  service)^ 

•  Hours  devoted  to  scheduled  vs.  unscheduled  maintenances 

•  Average  cycle  time  for  unscheduled  maintenance  requests^ 

•  Use  of  technology  to  improve  efficiency  of  processes^ 

Real  Estate 

•  Gross  square  feet  owned  vs.  leased^ 

•  Square  feet  per  employee^ 

•  Real  estate  cost  (total,  per  employee,  and  per  square  foot)^ 

•  Real  estate  cost  as  a  percentage  of  net  revenues^ 

•  Real  estate  assets  as  a  percentage  of  equity^ 

Safety 

•  Worker’s  compensation  claims^>l 

•  Safety  incidents^>l 

•  Number  of  lost  work  days  as  a  result  of  work  accidents^ 

•  Quality  of  environmental  and  safety  programs^>l 

Human  Resources 

•  Employee  turnover  rate  (cumulative  employees  for  the  period/total 
number  of  positions  authorized)^ 

•  Security  personnel  turnover  rate^ 

•  Custodial  employee  turnover  rate^ 

•  Training  hours  (total  and  hours  per  employee)s4 

•  Number  of  equal  employment  opportunity  suits! 
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Special  Interest 

•  Percentage  of  business  done  with  small  or  minority-  or  women- 

owned  businesses^ 


SOURCES: 

(a)  Presentation  by  David  Hartings,  Hewlett-Packard,  ‘TQRDCEB,” 
Tradeline  Conference,  San  Diego,  CA,  December  1997. 

(b)  Presentation  by  Thomas  Bogle  and  Pat  Shreffler,  CB/Koll,  “Customer 
Service  Call  Centers  for  Facilities  Organizations:  A  Strategic  Imperative,” 
Tradeline  Conference,  San  Diego,  CA,  December  1997. 

(c)  Presentation  by  Gary  Guarisco  and  Lloyd  Lichtenberg,  COMPASS 
Management  and  Leasing,  “Customer  Satisfaction  Measures  and  How  to  Use 
Them  to  Improve  Services,”  Tradeline  Conference,  San  Diego,  CA,  December 
1997. 

(d)  Presentation  by  David  Rodriguez,  Bellcore,  “Bellcore  Contracts  to  Get  an 
On-line,  Computerized  Measurement  Program,”  Tradeline  Conference,  San 
Diego,  CA,  December  1997. 

(e)  Presentation  by  Stephanie  Walker,  AmSouth  Bank,  “Strategic  Use  of 
Performance  Measures,”  Tradeline  Conference,  San  Diego,  CA,  December 
1997. 

(f)  Presentation  by  Paul  Garity,  KPMG  Peat  Marwick,  “Linking  Co^orate 
Real  Estate  Strategy  to  Performance  Measurement  and  Benchmarking,” 
Tradeline  Conference,  San  Diego,  CA,  December  1997. 

(g)  Presentation  by  Helen  R.  Arnold  and  Arthur  P.  Hall,  Cushman  & 
Wakefield,  “Benchmarking  as  Operational  Necessity,”  Tradeline  Conference, 
San  Diego,  CA,  December  1997. 

(h)  Presentation  by  Janet  Halo  and  Michael  Torre,  Merrill  Lynch,  “The 
Movement  of  Headquarters  Facilities  to  Lower  Cost  Environments,” 
Tradeline  Conference,  San  Diego,  CA,  December  1997. 

(i)  Presentation  by  Jim  Carmody,  Jeff  Mervin,  and  Leo  Michaud,  Morrison 
Knudsen,  Operations  and  Maintenance  Division,  “FM  Costs:  What  They 
Should  Be  and  How  to  Get  Them  Lower,”  Tradeline  Conference,  San  Diego, 
CA,  December  1997. 
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(j)  Presentation  by  Greg  Myers,  Intel,  “Putting  Performance  Numbers  on 
High-Tech  Facilities — Intel’s  Approach,”  Tradeline  Conference,  San  Diego, 
CA,  December  1997. 

(k)  Open  Forum,  Tradeline  Conference,  San  Diego,  CA,  December  1997. 

(l)  Literature  provided  by  customers  and  suppliers  during  interviews. 
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